{"id":"65611438-2252-48ad-b3ca-e84465eb05bf","arxiv_id":"2505.16845","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A neural speech codec that dynamically varies frame rate per segment using waveform entropy achieves competitive or better reconstruction quality at lower average frame rates than constant frame rate baselines.","lead":"This paper introduces Temporally Flexible Coding (TFC), a method that lets a neural speech codec vary its frame rate over time, using more codes for information-dense speech and fewer codes for silence. The authors show the codec keeps or improves reconstruction quality while reducing the total number of codes, which could speed up downstream speech generation models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The VFR-vs-CFR comparison may not hold average bitrate and frame rate fixed: the paper's own example ratios (r_f=0.2, r_m=0.7, r_c=0.1) imply 43.125 Hz and 3.45 kbps, not 37.5 Hz/3 kbps.","rationale":"The reader's conditional verdict is driven by the unvalidated entropy router and the model's generalization to arbitrary granularity ratios. My stress test finds a more direct threat: the paper's own numeric example for the 'average 3kbps/37.5Hz' condition does not satisfy its definition of average frame rate. If Figure 2a uses similar ratios, the VFR curves run at higher average frame rate and bitrate than the CFR baseline, so the improvement would be expected and the central claim would be untested. This is an internal inconsistency, not a disagreement with consensus. The concrete test of recomputing R and B from the exact experimental ratios would settle it. I therefore keep the verdict conditional (or would move it to reject if the Figure 2a ratios are confirmed unmatched), and I partially agree with the reader: the entropy-router uncertainty is real, but the rate-control issue is more fundamental because it determines whether the headline comparison is valid at all.","tokens_in":8595,"tokens_out":14801,"duration_ms":116860,"concrete_test":"Extract from the paper's code or supplementary material the exact granularity ratios used for each marker in Figure 2a, and recompute the average frame rate R = 75*r_f + 37.5*r_m + 18.75*r_c and bitrate B = 80*R. Verify that every VFR marker satisfies R = 37.5 Hz and B = 3 kbps, equivalently 3*r_f + r_m = 1. If any marker deviates, replot the VFR and CFR curves at exactly matched R and B; if the gains vanish, the central claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim in Section 4.3 requires a comparison at fixed average bitrate and average frame rate. The paper states it uses 'an average 3kbps bitrate & 37.5Hz frame rate setting' and gives as an example granularity ratios r_f=0.2, r_m=0.7, r_c=0.1. But from Section 3.2, the average transmitted frame rate is F*r_f + (F/2)*r_m + (F/4)*r_c = 75*0.2 + 37.5*0.7 + 18.75*0.1 = 43.125 Hz, and with N_q=8 codebooks of 10 bits each this yields 3.45 kbps, not 3 kbps. Thus the illustrative VFR condition is not rate-matched to the CFR 37.5Hz/3kbps baseline. If the Figure 2a points were generated from similarly unmatchable ratios, the reported 'consistent improvement' may simply reflect a higher average bitrate. The paper does not provide the actual ratios or per-point bitrates for Figure 2a, nor release code. Even setting aside whether waveform entropy is the correct router, the headline VFR-versus-CFR comparison is not shown to be controlled.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Temporally Flexible Coding (TFC), a plug-and-play module that introduces variable frame rate (VFR) into neural speech codecs built on a DAC backbone. TFC computes a non-parametric temporal entropy from raw waveform amplitudes and uses quantile-based masks to assign each temporal segment one of three granularities (75, 37.5, or 18.75 Hz), with the average frame rate controlled by user-supplied granularity ratios. The decoder fuses the three resolutions through a conditional hierarchical design. Experiments compare DAC+TFC with the DAC baseline in constant frame rate (CFR) mode at several bitrates, and compare VFR against CFR at a nominal average of 3 kbps / 37.5 Hz. The paper reports improved reconstruction metrics for TFC in both comparisons and claims that a single model can span average frame rates from 18.75 to 75 Hz while preserving quality.","tokens_in":8924,"tokens_out":5838,"duration_ms":36813,"significance":"If the VFR-versus-CFR comparison is properly controlled, the paper addresses a real gap: existing neural codecs vary bitrate through codebook dropout but still emit a fixed number of frames per second, which is suboptimal for downstream autoregressive models. The TFC design is simple, loss-agnostic, and orthogonal to other low-frame-rate codec efforts, so the core idea has clear practical value. Credit is due for the explicit CFR/VFR comparison, the use of multiple objective and perceptual metrics, and the attempt to connect bitrate and sequence-length reduction. The main significance hinges on whether the reported VFR gains are measured at a truly fixed average bitrate and average frame rate; the current manuscript does not yet demonstrate this.","major_comments":[{"comment":"The illustrative granularity ratios r_f=0.2, r_m=0.7, r_c=0.1 do not correspond to the stated average 3 kbps / 37.5 Hz setting. With the F=75 Hz backbone, the average number of transmitted frames per second is 75*0.2 + 37.5*0.7 + 18.75*0.1 = 43.125 Hz, and with N_q=8 10-bit codebooks this gives 43.125*80 = 3450 bps, not 3000 bps. Thus the example VFR configuration is not rate-matched to the 37.5 Hz / 3 kbps CFR baseline. The paper must report the exact granularity ratios and the realized average bitrate and frame rate for every point in Figure 2a, and either correct the example or confirm that the plotted points use ratio sets that genuinely average to 37.5 Hz. Without this, the 'consistent improvement' in Figure 2a could be explained by a higher average bitrate.","section":"Section 4.3 / Section 3.2"},{"comment":"No evaluation corpus, utterance count, or confidence intervals are reported, and the WER numbers are internally anomalous. In Table 1, DAC at 6 kbps has WER 4.125, which is worse than DAC at 3 kbps (2.443) and worse than DAC+TFC at 3 kbps (2.193); this is surprising and unexplained. Since WER is one of the two metrics used in Figure 2b, a single ASR run without error bars cannot support the claim that TFC configurations 'consistently outperform' the 75 Hz baseline. Please specify the test set, report error bars or significance tests, and discuss the WER anomalies.","section":"Section 4.3 / Figure 2 / Table 1"},{"comment":"The entropy-based router is not validated. The Gaussian affinity entropy depends on the unstated sigma and bin count N, and the paper provides no evidence that raw-waveform amplitude entropy tracks codec reconstruction difficulty. The VFR-versus-CFR comparison shows that mixing granularities helps under the current training recipe, but it does not establish that entropy-based routing, rather than the mixture of granularities itself, is responsible. An ablation comparing entropy routing with uniform or random routing at the same granularity mixtures, or a correlation analysis between H(T) and per-segment reconstruction error, is needed to support the causal interpretation in the abstract and Section 3.1.","section":"Section 3.1 / Eq. (2)-(4)"},{"comment":"The model is trained only with granularity ratios r_f=0.4, r_m=0.3, r_c=0.3, but inference advertises arbitrary ratios (e.g., 0.2/0.7/0.1 and the varying ratios in Figure 2b). The paper does not demonstrate that the model generalizes across the full claimed ratio range; if part of the VFR gain comes from matching the training distribution of ratios, the flexibility claim is overstated. Please provide a ratio-wise breakdown of results or otherwise show that performance is stable across the 18.75-75 Hz average-frame-rate range.","section":"Section 4.1 / Section 4.2 / Section 4.3"}],"minor_comments":[{"comment":"The values of sigma, N, and epsilon in Eqs. (2)-(4) are never reported, and no sensitivity analysis is given; please state the hyperparameters used in the reported experiments.","section":"Section 3.1 / Section 4.1"},{"comment":"The paper claims to introduce VFR to neural speech codecs 'for the first time' but Section 2.2 discusses VFR in semantic tokens from self-supervised models; please qualify the claim as first in acoustic neural codecs.","section":"Abstract / Section 1 / Section 2.2"},{"comment":"The figure axes and the meaning of '75Hz granularity' on the x-axis are not defined in the text; please clarify how each plotted point is generated and how the average frame rate and bitrate are held fixed.","section":"Section 4.3 / Figure 2"},{"comment":"The 6 kbps DAC row appears to have a formatting issue where the Nq value is merged with the Mel distance; please fix the table layout.","section":"Table 1"},{"comment":"The sentence 'Figure 2a obviously shows...' is informal; please replace 'obviously' with a quantitative statement of the margin and its variability.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like an extended abstract than a full journal paper: it reports a single backbone, no code release, no ablations for the entropy router, and no error bars. The VFR idea is timely and the TFC module is simple, but the rate-matching arithmetic in Section 4.3 must be checked before the central claim can be accepted. I would not reject on novelty grounds; the issue is empirical support and reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe real news here is the first VFR acoustic codec, and the mechanism is clean: hierarchical multi-resolution latents with entropy-quantile masks. The CFR comparison in Table 1 is the strongest part — at equal bitrate, dropping from 75Hz to 37.5Hz with the same total codebook budget actually improves quality on most metrics. That alone is worth a look.\n\nBut the headline VFR-vs-CFR comparison in Section 4.3 has a numerical problem. The paper says it uses 'an average 3kbps bitrate & 37.5Hz frame rate setting' and gives example ratios r_f=0.2, r_m=0.7, r_c=0.1. With F=75, those ratios give 75*0.2 + 37.5*0.7 + 18.75*0.1 = 43.125Hz, and with 8 codebooks at 10 bits that is 3.45kbps, not 3kbps/37.5Hz. So the illustrative configuration is not rate-matched to the CFR baseline. If the plotted VFR points in Figure 2a were generated from similarly unmatched ratios, the reported improvement could just be extra bitrate. The paper doesn't give the actual ratios or per-point bitrates for the figure, so we can't tell. This needs to be fixed before the central claim is credible.\n\nOther soft spots are more routine. The evaluation set is never named, there are no error bars or significance tests, and the WER numbers behave oddly: DAC at 6kbps has worse WER (4.125) than at 3kbps (2.443), which suggests either noise or some confound. The entropy router (Eq. 2-4) depends on unreported sigma and bin count N, and there's no ablation showing that high-entropy regions are actually where fine-grained codes help. The model is trained with fixed granularity ratios (0.4/0.3/0.3) and assumed to generalize to arbitrary inference ratios; the example ratio in the paper is far from that training distribution, which adds to the concern.\n\nCredit where due: the paper is honest about its limitation — one backbone, no downstream evaluation — and the VFR formulation is genuinely new relative to the VRVQ line. This deserves a serious referee. A competent reviewer can ask for the rate-matching numbers, a named test set with error bars, and a small ablation on the entropy router. If those come back clean, the result is likely real.\n\nRecommendation: send to peer review, with the arithmetic fix required before acceptance.","headline":"VFR for acoustic codecs is a real idea, but the key comparison has an arithmetic slip and the eval reporting is too thin to accept as-is.","tokens_in":9458,"tokens_out":4060,"would_cite":false,"duration_ms":29479,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural speech codec that varies frame rate by information density beats constant-rate coding at the same bitrate.","keywords":["neural speech codec","variable frame rate","temporal entropy","bitrate control","speech compression","RVQ","codec language model","sequence length reduction"],"falsifier":"Take a set of evaluation utterances and replace TFC's entropy-based mask with an oracle mask that, at each average frame rate, assigns fine frames to the windows where the fine-versus-coarse reconstruction error difference is largest, then compare the resulting quality scores; if the oracle mask is noticeably better, waveform entropy is not tracking coding difficulty.","tokens_in":8417,"feed_emoji":"🎙️","tokens_out":6292,"duration_ms":48593,"temperature":0.7,"pith_summary":"The paper tries to establish that a neural speech codec can operate at a variable frame rate (VFR), rather than the standard constant frame rate (CFR), and that this flexibility improves reconstruction quality rather than merely saving bits. The proposed Temporally Flexible Coding (TFC) estimates the information density of each short speech segment with a simple non-parametric entropy calculation, then routes quiet or predictable segments to coarse, low-frame-rate codes and busy segments to fine, high-frame-rate codes. Built on the DAC backbone for 24 kHz audio, TFC can be set at inference time to any average frame rate between 18.75 Hz and 75 Hz by choosing granularity ratios. The paper reports that at a fixed average bitrate and average frame rate, TFC beats constant-rate DAC on most quality metrics, and that even at lower frame rates than the 75 Hz baseline it matches or improves perceptual quality. If true, this would let speech codecs and the generative models built on them run with shorter token sequences and tunable latency without sacrificing audio quality.","feed_headline":"Variable frame rate codec beats constant rate at same bitrate","feed_subtitle":"TFC lets one model run at 18.75–75 Hz, using more frames on busy speech and fewer on silence.","key_machinery":"The load-bearing mechanism is the entropy-routed granularity mask. TFC computes a per-window Shannon entropy $H(T)$ from Gaussian affinities of raw waveform amplitudes to a set of bins, then derives binary masks $b_c\\, b_m\\, b_f$ at coarse, medium, and fine resolutions by comparing those entropy sequences to quantiles specified by the granularity ratios $r_c, r_m, r_f$. These masks decide which temporal windows are coded at which resolution; the chosen quantized vectors are repeated and merged by element-wise mask multiplication to form a single latent, and a conditional hierarchical decoder progressively refines coarse features into fine ones. The masks make frame-rate allocation a deterministic, user-controllable function of the signal's measured information density.","core_discovery":"The paper's central claim is that entropy-guided variable frame rate allocation yields better audio reconstruction than constant frame rate operation at the same average bitrate and average frame rate, and it also shortens the token sequence. The method, TFC, creates three temporal resolutions from the encoder's latent: fine frames at 75 Hz, medium at 37.5 Hz, and coarse at 18.75 Hz. For each window, a non-parametric Shannon entropy computed on raw waveform amplitude determines which resolution is used, via quantile thresholds set by user-chosen granularity ratios. Quantized vectors from the chosen resolution are repeated and fused into a single sequence aligned with the finest grid, and a conditional hierarchical decoder refines the signal from coarse to fine. In the reported tables and figures, DAC+TFC outperforms the DAC baseline across most metrics, and mixing frame rates at the same average rate yields consistent gains over pure 37.5 Hz or the 75 Hz constant-rate baseline, supporting the claim that time-varying frame rates are beneficial.","pith_inferences":["The paper trains only with granularity ratios $(0.4, 0.3, 0.3)$; a natural test is whether retraining on the actual inference ratio distribution changes the gap to CFR, which would reveal how much of the gain comes from the routing rule versus the added capacity of the multi-resolution decoder.","The entropy signal is computed on raw amplitude; using the codec's own reconstruction error or a learned importance map as the routing signal is a direct alternative that could be compared against the entropy mask.","Because TFC changes frame rate, it should also cut the computational cost of decoding and the memory footprint of autoregressive decoding (KV cache) beyond the token-count reduction, though the paper does not measure this.","VFR allocation could be applied to semantic tokens from self-supervised models, where the entropy signal may be even stronger, potentially yielding very low average frame rates for speech language models."],"forward_implications":["A single trained TFC codec can span average frame rates from 18.75 Hz to 75 Hz by changing the granularity ratios at inference, replacing the need for multiple codecs or for codebook dropout as the only bitrate knob.","At a fixed average bitrate and average frame rate, VFR allocation improves Mel distance, STFT distance, UTMOS, STOI, and WER relative to constant frame rate operation, as shown in the paper's Figure 2a and Table 1.","Lowering the frame rate to 56.25 Hz or below can still match or exceed the 75 Hz constant-rate baseline on perceptual metrics, which implies the token sequence can be shortened without hurting quality.","Shorter token sequences at a given bitrate mean faster autoregressive generation for speech language models, since generation speed depends on frame count, not codebook count.","TFC is designed to be orthogonal to other low-frame-rate codec techniques and can be stacked with them."],"supporting_citations":[{"why":"Provides the DAC backbone codec whose 75 Hz latent TFC reuses, along with the official 24 kHz configuration and training setup.","marker":"[3]"},{"why":"Supplies the dynamic granularity adaption idea and the conditional hierarchical decoder design that TFC adapts from image compression to speech.","marker":"[17]"},{"why":"Establishes the earlier use of entropy-based variable frame rate sampling for speech in ASR, motivating the use of temporal entropy in this paper.","marker":"[18]"},{"why":"Supports the choice of entropy-based routing over a learned neural router, which the authors cite as less stable and worse-performing in image coding.","marker":"[28]"},{"why":"Introduces variable bitrate in RVQ audio codecs but keeps constant frame rate, providing the baseline that motivates the need for VFR.","marker":"[16]"},{"why":"Introduces quantizer dropout, the standard CFR bitrate-control mechanism that TFC compares against and uses with Nq=8 in inference.","marker":"[1]"},{"why":"Defines the 75 Hz high-frame-rate neural codec (EnCodec) against which low-frame-rate and VFR designs are motivated.","marker":"[2]"}],"fun_headline_variants":["Entropy-guided frame rates improve speech codec at same bitrate","Variable frame rate codec trims tokens without losing quality","TFC: adaptive frame rates for richer speech at lower bitrate","Smarter frames: variable rate codec beats fixed rate on speech"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the entropy of raw waveform samples in a short window measures how many code frames that window needs for good reconstruction, so routing low-entropy windows to coarse frames will not hurt quality.","fun_headline_variants_meta":{"raw":{"variants":["Entropy-guided frame rates improve speech codec at same bitrate","Variable frame rate codec trims tokens without losing quality","TFC: adaptive frame rates for richer speech at lower bitrate","Smarter frames: variable rate codec beats fixed rate on speech"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1381,"prompt_tokens":905,"completion_tokens":476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":403}},"tokens_in":521,"tokens_out":476,"duration_ms":3864,"temperature":1.0,"reasoning_tokens":403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:53:36.770819+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of evaluation utterances and replace TFC's entropy-based mask with an oracle mask that, at each average frame rate, assigns fine frames to the windows where the fine-versus-coarse reconstruction error difference is largest, then compare the resulting quality scores; if the oracle mask is noticeably better, waveform entropy is not tracking coding difficulty.","supporting_citations":[{"cited_title":"diffusing","cited_arxiv_id":null,"evidence_quote":"Provides the DAC backbone codec whose 75 Hz latent TFC reuses, along with the official 24 kHz configuration and training setup."},{"cited_title":"Fast and high- quality auto-regressive speech synthesis via speculative decod- ing,","cited_arxiv_id":null,"evidence_quote":"Introduces variable bitrate in RVQ audio codecs but keeps constant frame rate, providing the baseline that motivates the need for VFR."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces quantizer dropout, the standard CFR bitrate-control mechanism that TFC compares against and uses with Nq=8 in inference."}],"review_version":1}