{"id":"76ecd39e-2c49-4b51-8e5f-ac188b03affb","arxiv_id":"2506.00800","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":6,"one_line_summary":"CLAP-ART improves automated audio captioning by feeding BART discrete tokens produced from a semantic audio representation (BEATs) rather than from a waveform codec.","lead":"This paper swaps the audio compression tokens used by an automated captioning system for tokens meant to carry sound meaning instead of just waveform information. The result is CLAP-ART, which beats the prior EnCLAP approach on two standard audio captioning benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CLAP-ART's semantic-token advantage is not isolated from token sequence length and codebook training; equal-length controls are needed before attributing the gains to semantics.","rationale":"The stress-test pass finds one structurally weak point in the empirical argument. The paper's headline comparison (Table 1) changes the source representation, the token rate, and the codebook learning rule simultaneously. Because Section 4.1 flattens BEATs patches into FT vectors while EnCLAP uses L EnCodec time tokens, the CLAP-ART input sequence can differ substantially in length and structure; because BEATs iter3+ is supervised with AudioSet and AudioCaps is a subset of AudioSet, part of the gain may reflect domain overlap rather than a general property of semantic-rich tokens. This is not an internal inconsistency, and the w/o-CLAP ablation (Table 2) is a real point in the paper's favor: CLAP-ART without the CLAP embedding still beats EnCLAP with the CLAP embedding, so the ART tokens carry usable information independent of CLAP. The remaining issue is attribution rather than existence of a gain. The proposed equal-length and same-pipeline controls would resolve whether the mechanism is semantic richness or the tokenizer configuration. Since the reader already conditioned the verdict on this cluster of concerns, the verdict should remain conditional; the paper should not be accepted as-is without such a control or, at minimum, an explicit statement that the central claim concerns the full BEATs-RVQ tokenizer rather than semantic content alone.","tokens_in":9718,"tokens_out":6377,"duration_ms":63513,"concrete_test":"On AudioCaps, rerun CLAP-ART(BEATs-RVQ) with token sequence length equalized to EnCLAP: average-pool or strided-sample the BEATs-RVQ tokens to the EnCodec temporal resolution before BART embedding, keeping 16 RVQ layers and codebook size 1024; in parallel, run EnCLAP with EnCodec tokens upsampled or repeated to match the BEATs-RVQ sequence length. If the SPIDEr gap (49.8 vs 46.3 in Table 1) collapses below about 1 point in either arm, the semantic interpretation is not supported. A stronger second arm is to apply the same RVQ procedure to a non-semantic baseline feature set, such as a randomized projection of a spectrogram, with matched sequence length and training recipe; if that baseline also beats EnCLAP, the benefit is not attributable to semantics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that quantizing semantically rich BEATs features into discrete tokens is what improves AAC over EnCodec tokens. The evidence for this is the Table 1 comparison, which varies three things at once: the source representation (BEATs vs EnCodec), the token sequence length, and the codebook training procedure. In Section 4.1, BEATs-RVQ flattens the BEATs encoder output into FT 768-dimensional vectors, producing a sequence of length FT; EnCLAP uses L time-frame EnCodec tokens (Eqs. 1-2), and EnCodec codebooks are learned end-to-end for waveform reconstruction while BEATs-RVQ codebooks are fit offline by k-means on the training set. The +3.5 SPIDEr gain on AudioCaps could therefore come from a longer token sequence, the different codebook capacity or training objective, or the fact that AudioCaps is a subset of the AudioSet labels used to supervise BEATs, rather than from 'semantic richness' per se. The w/o-CLAP ablation (Table 2) and the monotone improvement with RVQ layers (Table 3) are genuine supporting evidence, and BEATs-AT's single-layer gain reduces the length confound somewhat, but no experiment keeps the tokenization pipeline fixed while varying only the semantic content of the source representation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes CLAP-ART, an automated audio captioning method that feeds BART with discrete tokens derived from semantically rich audio representations: BEATs-AT, which naively uses BEATs' acoustic tokenizer, and BEATs-RVQ, which applies residual vector quantization to BEATs' audio encoder features. These tokens are concatenated with a CLAP audio embedding and used to fine-tune BART, following the EnCLAP framework. The authors report that CLAP-ART outperforms EnCLAP on AudioCaps and Clotho across all evaluated metrics, with ablations removing the CLAP embedding and varying the number of RVQ layers. The paper's central hypothesis is that semantic-rich discrete tokens are more beneficial for AAC than reconstruction-oriented EnCodec tokens.","tokens_in":9982,"tokens_out":7720,"duration_ms":74471,"significance":"If the causal attribution were established, this would be a useful design principle for AAC: tokenizers should preserve semantic content rather than waveform fidelity. The paper has genuine strengths: it uses six random seeds with standard deviations, reproduces the EnCLAP baseline, includes a w/o-CLAP ablation that directly tests the informativeness of the discrete tokens, and compares against SpeechTokenizer- and HuBERT-based alternatives. These make the main comparative result—that BEATs-derived tokens help—fairly robust. However, the title and conclusions make a stronger causal claim about semantic richness than the current experiments isolate, because the central comparison varies several factors at once.","major_comments":[{"comment":"The central attribution of gains to 'semantic richness' is confounded with token sequence length and codebook training. In Section 4.1, BEATs-RVQ flattens the BEATs encoder output into F*T 768-dimensional vectors, whereas EnCLAP's EnCodec tokens have a time length L in Eq. (1); these lengths are not matched. Additionally, the EnCodec codebooks are learned end-to-end for reconstruction, while the BEATs-RVQ codebooks are obtained by k-means on the training features. The +3.5 SPIDEr gain in Table 1 could therefore be explained by differences in input sequence length, codebook capacity, or codebook learning objective rather than by semantic content per se. A control that varies only the source representation while keeping the tokenizer pipeline, sequence length, and codebook size fixed (for example, applying the same RVQ/k-means procedure to EnCodec encoder features) is needed to support the title's causal claim.","section":"Sec. 4.1 and Table 1"},{"comment":"The paper acknowledges that BEATs iter3+ was pre-trained on AudioSet and that AudioCaps is a subset of AudioSet, but it does not treat this as a threat to the central claim. The largest improvements are on AudioCaps (SPIDEr +3.5 for BEATs-RVQ in Table 1), while the Clotho gains are modest (+1.3), and the Clotho experiments are pre-trained on AudioCaps (Section 5.1), so they do not remove the overlap. Because the hypothesis concerns general sounds, the authors should either evaluate on a benchmark disjoint from the AR pre-training data or use an AR model whose pre-training does not include labels from the evaluation domain; at minimum, the conclusion should be tempered to acknowledge that the observed benefit may partly reflect AudioSet-label leakage.","section":"Sec. 5.2"},{"comment":"The claim that increasing the number of RVQ layers improves performance by packing more semantic information is based only on mean scores in Table 3, with no standard deviations or significance tests, although six seeds were used for the other tables. The differences between adjacent settings are small (e.g., SPIDEr 49.7 for N=8 vs. 49.8 for N=16 in Table 3), and the trend is not strictly monotonic (SPIDEr 48.8 for N=2 vs. 48.6 for N=4). Reporting the six-seed variability would make it possible to assess whether the multi-layer advantage is reliable, and the conclusion should be scaled to what the data support.","section":"Sec. 5.4 and Table 3"},{"comment":"The paper's hypothesis has two parts: the tokens should be semantic-rich and discrete. The experiments replace EnCodec tokens with BEATs-derived tokens, but they never compare BEATs-RVQ discrete tokens against continuous BEATs features in the same BART fine-tuning framework. Such a comparison is needed to show that quantization itself contributes to the improvement; without it, the gains could be due to the BEATs features alone, and the claim that discrete tokens are beneficial for BART would not be isolated. The w/o-CLAP ablation in Table 2 shows that the ART tokens alone carry information, but it does not test the continuous-input alternative.","section":"Secs. 3 and 5.2, Eqs. (3)-(4)"}],"minor_comments":[{"comment":"Equation (4) contains a typo: 'ICLAP-AR T' should be 'I_{CLAP-ART}' or similar.","section":"Eq. (4)"},{"comment":"In Section 4.1, the notation 'F × T' for patch dimensions and then 'length F T' is easy to confuse with the time length T used elsewhere; consider renaming the patch dimensions to avoid the clash.","section":"Sec. 4.1"},{"comment":"In Table 4 and the surrounding text, the 'Reference' rows are the best of six runs and are clearly labeled, but the sentence suggesting the result is 'comparable to the use of LLM' should explicitly remind the reader that LOAE's score comes from the original paper and is not averaged over seeds.","section":"Sec. 5.5"},{"comment":"The manuscript does not provide a link to an implementation of CLAP-ART; releasing code and checkpoints would strengthen the reproducibility of the reported six-seed results.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid empirical contribution, but the causal claim about semantic richness needs stronger controls. If the authors can add a matched-length/RVQ-pipeline control, a disjoint-domain evaluation, standard deviations for Table 3, and a continuous-feature baseline, the paper would be suitable for publication. I see no circularity or fabrication concerns; the issues are about experimental isolation, not about the validity of the reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid incremental method paper. The genuinely new bit is BEATs-RVQ — k-means residual vector quantization on BEATs encoder features, fed to BART alongside a CLAP embedding — and it beats a reproduced EnCLAP baseline on AudioCaps and Clotho across all metrics, with six-seed means and standard deviations. That is real evidence. The w/o-CLAP ablation is the most convincing part: CLAP-ART(BEATs-RVQ) without the CLAP embedding still lands near EnCLAP with CLAP, which says the discrete tokens carry a lot of the semantic load. The monotone improvement as RVQ layers go from 1 to 16 supports the multi-layer story. Credit where due: the experimental discipline is better than most in this subfield.\n\nThe soft spots are real but addressable. The paper's central claim is that \"semantic richness\" is what drives the gain, yet Table 1 varies three things at once: source representation, token sequence length, and codebook training procedure. BEATs-RVQ flattens F×T patches into FT vectors of dimension 768, while EnCodec has its own frame-rate; the codebooks are also fit by k-means offline rather than learned end-to-end for reconstruction. The BEATs-AT and N=1 comparisons reduce the length confound partially, but no experiment fixes the tokenization pipeline and swaps only semantic content. Also, AudioCaps is a subset of AudioSet, which supervised BEATs, so some of the AudioCaps gain could be domain overlap. The paper acknowledges this in passing, which is fair, but it undermines the strong \"semantic-rich\" phrasing in the title and abstract.\n\nMinor issues: no code release despite a reproducibility pledge; the Clotho learning rate and pre-training epochs come from an unreported grid search; and Table 4 reports a best-of-six \"reference\" run alongside averaged runs, which makes the claimed parity with the 7B-parameter LOAE system softer than the surrounding text suggests. The citation pattern looks fine — EnCLAP, BEATs, SpeechTokenizer, and SemantiCodec are all relevant and credited.\n\nWho this is for: researchers working on AAC or on discrete-token interfaces to language models. It deserves a serious referee. My own verdict would be conditional acceptance, asking for an equal-token-length control, clearer separation of averages from best runs, and language that matches the actual scope of the evidence. Recommendation: send it to peer review; the concerns are fixable and the contribution is legitimate.","headline":"Useful, honest incremental AAC result; the semantic-token story is plausible but not fully isolated from token length and codebook training.","tokens_in":10561,"tokens_out":1566,"would_cite":true,"duration_ms":16403,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feeding BART discrete tokens that encode general-sound semantics, obtained by vector-quantizing BEATs audio representations, improves automated audio captioning over codec-style waveform tokens.","keywords":["automated audio captioning","discrete token","residual vector quantization","audio representation","BEATs","EnCLAP","AudioCaps","Clotho"],"falsifier":"Run CLAP-ART and EnCLAP with matched token sequence lengths, for example by resampling BEATs-RVQ tokens to EnCodec's temporal rate; if the SPIDEr advantage disappears, the improvement is due to input format rather than semantic content.","tokens_in":9514,"feed_emoji":"🎧","tokens_out":10487,"duration_ms":84025,"temperature":0.7,"pith_summary":"This paper tries to establish that automated audio captioning improves when the language model receives discrete audio tokens that carry semantic content about general sounds, rather than tokens optimized only for reconstructing waveforms. The authors build CLAP-ART, which vector-quantizes features from the pretrained audio representation model BEATs and feeds the resulting tokens to BART together with a CLAP audio embedding. On the AudioCaps and Clotho benchmarks, CLAP-ART outperforms the EnCLAP baseline on every reported metric. If the claim is right, caption quality can be raised by choosing what the discrete input tokens represent, without extra training data or a larger language model.","feed_headline":"Semantic audio tokens beat waveform codecs for captioning","feed_subtitle":"CLAP-ART feeds BART discrete BEATs tokens, not EnCodec tokens, improving caption scores on AudioCaps and Clotho.","key_machinery":"The central mechanism is the audio representation tokenizer (ART), which converts continuous, semantically rich audio representations into discrete tokens. BEATs-RVQ, the main variant, takes 768-dimensional feature vectors from BEATs' audio encoder and passes them through 16 residual vector quantization layers with 1024-entry codebooks, projecting each vector to its nearest codebook entry and quantizing the residual at each layer. The resulting multi-layer discrete tokens are embedded and concatenated with a projected CLAP audio embedding to form the input sequence for BART.","core_discovery":"The central claim is that semantic-rich discrete tokens derived from semantically rich audio representations are better inputs than reconstruction-oriented codec tokens for fine-tuning a language model in automated audio captioning. CLAP-ART instantiates this by replacing EnCodec tokens with tokens from an audio representation tokenizer (ART): BEATs-AT uses BEATs' acoustic tokenizer directly, and BEATs-RVQ applies residual vector quantization to BEATs' audio encoder outputs, with k-means codebooks built on the training set. The experiments show CLAP-ART outperforming EnCLAP across METEOR, CIDEr, SPICE, SPIDEr, and FENSE on both AudioCaps and Clotho, and the ablation without the CLAP embedding degrades far less than EnCLAP's, supporting the attribution of the gain to the semantic content of the ART tokens.","pith_inferences":["An implication the paper leaves implicit is that the comparison does not hold token sequence length fixed: BEATs features are flattened into a long vector sequence while EnCodec tokens follow their own frame rate, so a matched-length experiment would separate the semantic-content effect from the input-format effect.","Because the BEATs model was trained with AudioSet supervision, a natural extension is to test CLAP-ART on audio domains outside AudioSet categories; larger gains on AudioCaps, a subset of AudioSet, suggest the benefit may depend on overlap between the representation model's pretraining domain and the captioning domain.","The RVQ codebooks are computed on the training set before fine-tuning; learning or adapting the codebooks jointly with the captioning objective could be a testable improvement that may further tune the tokens to the captioning task."],"forward_implications":["Replacing reconstruction-oriented EnCodec tokens with semantic quantized tokens from BEATs improves captioning on AudioCaps and Clotho across all five reported metrics.","ART discrete tokens alone, without the CLAP embedding, retain enough semantic information to reach captioning scores close to the full EnCLAP baseline, making discrete semantic tokens a strong standalone input.","Increasing the number of RVQ layers from 1 to 16 raises performance, indicating that multi-layer quantization packs more useful semantic information for caption generation.","Tokens derived from speech-focused models such as SpeechTokenizer and HuBERT-RVQ do not improve over EnCodec, so the relevant semantics are those of general sounds rather than speech phonemes or linguistic content.","With a BART-base model trained only on AudioCaps, CLAP-ART reaches SPIDEr scores near a 7-billion-parameter LLM-based method that uses extra data, which the paper reads as evidence that token representation choice matters as much as language-model scale."],"supporting_citations":[{"why":"Defines the EnCLAP baseline and the training framework that CLAP-ART extends and compares against.","marker":"[6]"},{"why":"Provides the BEATs audio encoder and acoustic tokenizer whose outputs ART quantizes.","marker":"[12]"},{"why":"Supplies the EnCodec discrete tokens used by the baseline and the residual vector quantization procedure adopted by BEATs-RVQ.","marker":"[15]"},{"why":"BART is the pretrained language model that CLAP-ART fine-tunes to generate captions.","marker":"[16]"},{"why":"CLAP supplies the audio-text joint embedding that CLAP-ART concatenates with the ART token sequence.","marker":"[14]"},{"why":"AudioCaps is one of the two captioning benchmarks on which CLAP-ART is trained and evaluated.","marker":"[23]"},{"why":"Clotho is the second benchmark, used after pretraining on AudioCaps.","marker":"[24]"},{"why":"SpeechTokenizer tokens serve as the speech-semantics comparison that does not improve captioning.","marker":"[18]"},{"why":"SoundStream is the other reference for the residual vector quantization steps used by BEATs-RVQ.","marker":"[20]"},{"why":"SemantiCodec provides evidence that EnCodec discrete tokens underperform AR features on general-sound downstream tasks, motivating the search for semantic tokens.","marker":"[17]"}],"fun_headline_variants":["Semantic audio tokens beat waveform codecs for captioning","CLAP-ART: Caption audio with semantic tokens, not codecs","Swap EnCodec for BEATs tokens to caption sound better","Audio captioning boosted by semantic-rich token inputs","Why semantic tokens outperform codec tokens in AAC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the BEATs features that survive vector quantization actually carry the general-sound semantics that improve caption generation, rather than the gain coming from other differences such as token sequence length or codebook design.","fun_headline_variants_meta":{"raw":{"variants":["Semantic audio tokens beat waveform codecs for captioning","CLAP-ART: Caption audio with semantic tokens, not codecs","Swap EnCodec for BEATs tokens to caption sound better","Audio captioning boosted by semantic-rich token inputs","Why semantic tokens outperform codec tokens in AAC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":2999,"prompt_tokens":872,"completion_tokens":2127,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":2044}},"tokens_in":488,"tokens_out":2127,"duration_ms":14313,"temperature":1.0,"reasoning_tokens":2044,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:57:04.194365+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CLAP-ART and EnCLAP with matched token sequence lengths, for example by resampling BEATs-RVQ tokens to EnCodec's temporal rate; if the SPIDEr advantage disappears, the improvement is due to input format rather than semantic content.","supporting_citations":[{"cited_title":"semantic-rich and discrete","cited_arxiv_id":null,"evidence_quote":"Defines the EnCLAP baseline and the training framework that CLAP-ART extends and compares against."},{"cited_title":"Automated audio cap- tioning by fine-tuning bart with audioset tags,","cited_arxiv_id":null,"evidence_quote":"Provides the BEATs audio encoder and acoustic tokenizer whose outputs ART quantizes."},{"cited_title":"Improving audio captioning models with fine- grained audio features, text embedding supervision, and llm mix- up augmentation,","cited_arxiv_id":null,"evidence_quote":"Supplies the EnCodec discrete tokens used by the baseline and the residual vector quantization procedure adopted by BEATs-RVQ."},{"cited_title":"Recap: Retrieval-augmented audio captioning,","cited_arxiv_id":null,"evidence_quote":"BART is the pretrained language model that CLAP-ART fine-tunes to generate captions."},{"cited_title":"Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multimodal research,","cited_arxiv_id":null,"evidence_quote":"CLAP supplies the audio-text joint embedding that CLAP-ART concatenates with the ART token sequence."},{"cited_title":"BART: Denoising sequence-to-sequence pre-training for natural language genera- tion, translation, and comprehension,","cited_arxiv_id":null,"evidence_quote":"AudioCaps is one of the two captioning benchmarks on which CLAP-ART is trained and evaluated."},{"cited_title":"Enhancing automated audio captioning via large language models with optimized audio encoding,","cited_arxiv_id":null,"evidence_quote":"SpeechTokenizer tokens serve as the speech-semantics comparison that does not improve captioning."},{"cited_title":"Audio Set: An ontology and human-labeled dataset for audio events,","cited_arxiv_id":null,"evidence_quote":"SoundStream is the other reference for the residual vector quantization steps used by BEATs-RVQ."}],"review_version":1}