{"id":"be687aa3-2013-47bb-afc5-02b65589c69b","arxiv_id":"2411.18447","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Injecting random noise into input embeddings during training lets purely autoregressive models generate continuous audio embeddings without quality degradation over long sequences.","lead":"A new training trick adds random noise to the input embeddings of autoregressive models that generate continuous audio embeddings, making them more robust to errors as they generate long sequences. On a music audio benchmark it beats several autoregressive and non-autoregressive baselines on Fréchet Audio Distance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (4) assumes inference errors are i.i.d. isotropic Gaussian mixtures, but actual autoregressive errors are temporally correlated and state-dependent; this unverified assumption is the load-bearing link.","rationale":"The reader's weakest_assumption correctly identifies Eq. (4) as the load-bearing link between the proposed training augmentation and the claimed robustness. My stress-test pass sharpens this into a concrete, testable concern: the assumed error model is implausible for autoregressive generation because errors are temporally correlated and state-dependent, yet the paper provides no direct measurement of the error distribution. The FADacc < FAD result is also confounded by context length differences between the first and second 10-second segments, as the paper itself notes. This does not disprove the method, and the controlled comparisons show consistent improvements, so the conditional verdict remains appropriate. However, the concerns are substantive enough that they should be resolved before accepting the stronger claim that CAM 'substantially outperforms' while avoiding error accumulation. I find no reason to demand a different verdict than CONDITIONAL; an unchanged verdict is therefore appropriate.","tokens_in":7118,"tokens_out":3077,"duration_ms":31142,"concrete_test":"Using a trained autoregressive baseline (e.g., MAR-RF) or a CAM variant without inference noise, perform teacher-forced generation on a validation set and collect the prediction errors e_t = x_t − x̂_t. Check three properties: (1) fit a multivariate Gaussian to e_t and compare its covariance to the identity matrix; (2) estimate the lag-1 autocorrelation of e_t across time steps; (3) compute marginal kurtosis to test Gaussianity. If the covariance is far from I, autocorrelation is significant, or kurtosis deviates from 3, then Eq. (4)'s error model is violated. A more decisive test: retrain CAM replacing the Gaussian ε in Eq. (4) with errors sampled from this measured teacher-forcing distribution, keeping k_t ∼ U(0,1).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is Eq. (4): at inference, generated embeddings are assumed to be expressible as x̃_t = k_t·ε + (1−k_t)·x_t, with ε ∼ N(0, I) and k_t ∼ U(0,1) drawn independently per timestep. This is the only explicit model of error accumulation in the paper, and training noise augmentation is built directly on it. Yet the paper never measures what accumulated errors actually look like. Real errors are generated by the model itself: each prediction error depends on the previous generated embeddings, propagates through the causal Backbone, and is therefore temporally correlated, state-dependent, and not generally isotropic Gaussian. Training with independent per-step convex mixtures of clean embeddings and fresh Gaussian noise can teach the Backbone to ignore arbitrarily corrupted inputs, but that is not the same as being robust to the specific compounding, correlated errors that appear at inference. The empirical evidence does not close this gap. FADacc = 0.394 being lower than FAD = 0.405 is also confounded: the first 10 s are generated from the SOS token with no prior context, while the second 10 s have a full 10 s of context, so the metric conflates warm-up/context effects with error accumulation. The paper even acknowledges this alternative explanation. Without a distributional check of Eq. (4), the claim that training-time noise augmentation 'makes the model robust against varying error levels at inference' rests on an unverified equivalence between injected i.i.d. Gaussian mixtures and the actual errors arising in the autoregressive loop.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Continuous Autoregressive Models (CAMs) trained with a noise-augmentation scheme to mitigate error accumulation when generating sequences of continuous embeddings in a purely autoregressive (GPT-style) manner. During training, input embeddings are corrupted as x̃_t = k_t ε + (1−k_t) x_t with k_t ∼ U(0,1) (Eq. 4), and at inference a small fixed noise level k_inf is added to generated embeddings before feeding them back to the backbone. Experiments on an internal music-audio latent dataset measure Frechet Audio Distance (FAD) on the first 10 seconds and FADacc on the next 10 seconds. The authors report FAD 0.405 and FADacc 0.394 for CAM, outperforming GIVT, MAR, and a non-autoregressive Rectified Flow baseline, and they show that GIVT also improves when trained with the proposed noise augmentation. The central claims are that noise augmentation makes continuous autoregressive models robust to error accumulation and that CAM preserves quality over extended sequences.","tokens_in":7466,"tokens_out":3021,"duration_ms":30199,"significance":"If the central claims hold, the paper makes a useful and practical contribution: it offers a simple, architecture-agnostic training recipe that enables GPT-style autoregressive generation on continuous embeddings, potentially unlocking KV-cache style efficiency and interactive audio generation. The study has notable strengths: the controlled comparisons use the same backbone architecture and roughly matched parameter counts across baselines, the proposed method is simple and easy to reproduce, and the transfer of the noise-augmentation benefit to GIVT (Fig. 2c) suggests the mechanism is not idiosyncratic to one model. However, the significance is currently limited by three load-bearing gaps: the assumed error model in Eq. (4) is not validated against actual inference errors, the inference noise level k_inf is tuned on the same evaluation metrics used for the headline comparison, and the FADacc metric conflates context length with error accumulation. These issues need to be addressed before the empirical claims can be considered robust.","major_comments":[{"comment":"The correctness of the training/inference mismatch argument rests on the assumption that accumulated prediction errors at inference are well modeled by x̃_t = k_t ε + (1−k_t) x_t with ε ∼ N(0, I) and independent k_t ∼ U(0,1). This is never validated. Actual autoregressive errors are produced by the model itself, depend on all previous generated embeddings, and are therefore temporally correlated, state-dependent, and not generally isotropic Gaussian. The paper should provide a diagnostic study comparing the distribution of real inference errors (e.g., per-step residual statistics, autocorrelation, Gaussianity tests) with the injected noise model, or at least an ablation that varies the noise model (e.g., correlated noise, non-Gaussian noise) to show the results are not overly sensitive to this choice. Without such evidence, the claimed equivalence between training-time corruption and inference-time errors is an unverified premise.","section":"Sec. 4, Eq. (4)"},{"comment":"The value k_inf = 0.02 used in all subsequent experiments is selected by the same FAD and FADacc metrics on which the final comparison is based. Because the evaluation set is also used for this model selection, the reported FAD/FADacc of 0.405/0.394 may be optimistically biased. The authors should either tune k_inf on a separate validation set and report the resulting test metrics, or provide error bars and significance tests across multiple independent evaluation draws. Additionally, the paper reports averages over 5 evaluations but no standard deviations; given that the difference between CAM and the best baseline (FAD 0.405 vs 0.442) is small, variance estimates are essential to establish that the improvement is not within noise.","section":"Sec. 5, Fig. 2b (Influence of Inference Noise)"},{"comment":"The claim that CAM 'does not demonstrate degradation when generating longer sequences' is not fully supported by FADacc because the metric compares two conditions that differ in context length, not only in accumulated error. The first 10 seconds are generated from the SOS token with no prior context, while the second 10 seconds are generated after a full 10 seconds of context. The paper itself acknowledges this confound (Fig. 2b discussion: 'the Backbone receives a maximum context of ~10 seconds, it generates all embeddings after the 10 seconds mark using a full context, which may result in higher quality embeddings'). To support the error-accumulation claim, the authors should provide a controlled comparison where context length is matched, for example by measuring FAD on independently generated 10-second segments conditioned on a fixed context, or by reporting per-segment quality as a function of generation length with equivalent conditioning.","section":"Sec. 5, FADacc definition and interpretation"},{"comment":"The comparison to the non-autoregressive baseline is limited to a single Rectified Flow model, and several experimental choices are underspecified: the number of denoising steps is selected per model from the range (10,100) 'that results in the lowest FAD' on the evaluation set, which again risks selection bias; no standard deviations or significance tests are reported; and the internal dataset is not available for independent verification. The authors should report confidence intervals, perform multiple training seeds for at least CAM and the closest baseline, and clarify whether the reported FAD numbers are best-of-N selections or averages over all evaluation draws. These additions are necessary to assess whether the improvement over MAR RF is statistically meaningful.","section":"Sec. 5, Evaluation and baselines"}],"minor_comments":[{"comment":"The notation for noise level is inconsistent: Eq. (2) uses t as the time variable in Rectified Flow, while Eq. (5) reuses σ_t for the noise level, and the sampler noise level is described as sampled from a lognormal distribution. This makes it hard to distinguish the RF time variable from the noise-level variable. Please use distinct symbols (e.g., s for RF time, σ for noise level).","section":"Sec. 3.2 and Eq. (5)"},{"comment":"Figure 2 is a single panel with three subfigures (a), (b), (c), but the caption does not describe each subfigure; the text refers to 'Tab. 2c' and 'Fig. 2b' inconsistently. Please separate the tables/figures or provide a clear caption that explains what each panel shows.","section":"Fig. 2"},{"comment":"The dataset is described only as an internal collection of ~20,000 single-instrument recordings. For reproducibility, the authors should provide more details about the music styles, the instrument distribution, and whether the dataset or any subset is publicly available. The Music2Latent autoencoder is also in-house; please specify the version and any relevant training details.","section":"Sec. 5, Datasets"},{"comment":"The baseline configuration details are incomplete: the exact number of denoising steps used for each diffusion-based model, the temperature and variance scaling for GIVT, and the training setup for 'GIVT+noise' are not fully specified. Please include a table with the final hyperparameter choices for every model.","section":"Sec. 5, Baselines"},{"comment":"There are several typographical and formatting issues, including inconsistent capitalization of 'VQ-VAE', missing spaces in citations (e.g., 'VQ-V AEs' in the Introduction), and a duplicated period in the References ([4] 'Tschannen, Cian Eastwood...'). A careful proofread would improve readability.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid workshop-level contribution with a simple and plausible method. The main concern is that the empirical evaluation, as presented, is not yet strong enough for a top-tier archival journal: the noise model in Eq. (4) is an unverified assumption, the inference noise level is tuned on the evaluation metric, and the headline 'no degradation over longer sequences' result is confounded by context length. These are fixable with additional experiments and statistical rigor, so I recommend major revision rather than rejection. The authors should also consider whether the claims about 'substantially outperforming' baselines hold after honest uncertainty quantification; the FAD margins are modest."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the core idea—train a causal transformer on noisy convex mixtures of input embeddings to simulate its own inference-time errors—is genuine and clean. The controlled comparisons use the same architecture and parameter counts, and the method beats MAR, MAR-RF, and GIVT variants on FAD, with a consistent gain and even lower FADacc. I also credit the GIVT+noise ablation: showing the same training trick helps a different AR baseline makes the mechanism less like a fluke. The writing is clear.\n\nThe soft spots start with the evidence base. No error bars or significance tests; FAD differences of 0.03–0.04 are modest, and with a single internal dataset and one evaluator, I can't tell if the gap is stable. Second, and more load-bearing, the assumed error model in Eq. (4) is never verified. They posit that accumulated errors are convex mixtures of clean embeddings and isotropic Gaussian noise, with mixing level uniform in [0,1]. But real errors are generated autoregressively—they are temporally correlated, state-dependent, and likely non-Gaussian. The stress-test note is right: training with independent per-step injected noise can make the Backbone tolerate arbitrary corruption, but that's not the same as robustness to the specific compounding errors at inference. A distributional check between injected noise and measured prediction errors would close the gap; without it, the central mechanism rests on an equivalence that is plausible but untested.\n\nThird, kinf=0.02 is selected on the same FAD/FADacc metrics that are then reported as evidence. That's a selection effect, not fatal, but it inflates confidence in the stated gains. Fourth, the FADacc < FAD result is confounded by context length—the first 10s have partial context, the second 10s have full context—and the paper admits this, so the 'no degradation' claim is weaker than advertised.\n\nWhat's genuinely new: applying scheduled-sampling-style noise augmentation to continuous-embedding autoregressive models, with the specific trick of not telling the Backbone the noise level. The omission of scheduled sampling and related robustness literature is a minor citation gap, not a flaw in the work.\n\nBottom line: this is a solid workshop-level result with a load-bearing assumption that needs checking. The idea will probably be built on. Send it to serious referees, but require the error-distribution validation, error bars, and ideally an external dataset or code release before the claims are accepted as stated.","headline":"Noise augmentation for continuous AR training is a genuine, clean idea with solid controlled comparisons, but the evidence base is thin enough that the headline claims outrun the measurements.","tokens_in":7944,"tokens_out":3240,"would_cite":true,"duration_ms":27817,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training on noise-augmented embeddings prevents error accumulation in continuous autoregressive models.","keywords":["continuous autoregressive models","error accumulation","noise augmentation","rectified flow","audio generation","latent embeddings","causal transformer","Fréchet Audio Distance"],"falsifier":"Record the actual prediction errors $e_t = \\hat{x}_t - x_t$ from a trained CAM on long generations and compare their distribution and temporal correlation with the assumed model $k_t \\varepsilon + (1-k_t)x_t$, $k_t \\sim U(0,1)$; if the empirical errors are correlated across timesteps or concentrated at certain mixing levels, training with the uniform schedule should underperform an alternative that uses the measured error distribution.","tokens_in":6910,"feed_emoji":"🎵","tokens_out":6692,"duration_ms":56208,"temperature":0.7,"pith_summary":"The paper claims that continuous autoregressive models degrade on long sequences because errors in predicted embeddings compound, and that this can be prevented by training the model on noisy mixtures of clean embeddings and Gaussian noise. The proposed augmentation, $\\tilde{x}_t = k_t \\varepsilon + (1-k_t) x_t$ with $k_t \\sim U(0,1)$, teaches the model to condition on inputs that resemble its own erroneous predictions. A small amount of inference-time noise further stabilizes generation. On musical audio, the method reports FAD 0.405 and FADacc 0.394, improving on all reported autoregressive and non-autoregressive baselines and showing no degradation over extended sequences.","feed_headline":"Noise trick keeps autoregressive audio models stable over long clips","feed_subtitle":"Noise-injected training lets a purely autoregressive audio model hold quality over long generations and beat baselines.","key_machinery":"The load-bearing object is the noisy-mixture input $\\tilde{x}_t = k_t \\varepsilon + (1-k_t) x_t$, where $k_t \\sim U(0,1)$ is sampled per timestep and hidden from the backbone. This forces the causal transformer backbone to act discriminatively, distinguishing real signals from error-like perturbations so that its conditioning $z_t$ stays informative. The sampler is trained with the Rectified Flow objective $L = \\mathbb{E}_t[\\|v_t - \\text{Sampler}(y_t|\\sigma_t, z_t)\\|^2]$, where $v_t = x_t - \\varepsilon$ and $\\sigma_t$ follows a lognormal schedule. At inference, adding the fixed noise level $k_{\\text{inf}} = 0.02$ to each generated embedding before re-feeding reduces the mismatch between training perturbations and actual prediction errors.","core_discovery":"The paper sets out to show that the error-accumulation failure of purely autoregressive models on continuous embeddings can be removed by a simple training-time noise augmentation. The authors model a generated embedding as $\\tilde{x}_t = k_t \\varepsilon + (1-k_t) x_t$ with $\\varepsilon \\sim \\mathcal{N}(0,I)$ and $k_t$ drawn uniformly from $[0,1]$, and feed these noisy mixtures to a causal backbone whose representation $z_t$ conditions a Rectified Flow sampler. The backbone is not told the value of $k_t$, so it must learn to separate real content from error-like signal. During inference, each generated embedding is lightly re-noised with $k_{\\text{inf}} = 0.02$ before being fed back. On musical audio, this yields FAD 0.405 and FADacc 0.394, with the long-sequence score matching or beating the first-sequence score and surpassing all reported baselines.","pith_inferences":["Editorial inference: the uniform schedule $k_t \\sim U(0,1)$ covers all mixing levels evenly, but generation errors may concentrate at small $k$ early in a sequence and grow with context; a curriculum that ties $k_t$ to positional or measured error statistics might improve training efficiency.","Editorial inference: because the paper does not measure the actual distribution of inference errors, the strongest test is to record $e_t = \\hat{x}_t - x_t$ and compare its marginals and autocorrelation to the assumed Gaussian-mixture form; if the match is poor, an error-adaptive augmentation should beat the uniform schedule.","Editorial inference: the finding that FADacc is lower than FAD may reflect the full 10-second context available after the first block rather than a general absence of error accumulation; isolating the context-length effect would clarify how much of the gain comes from noise augmentation alone.","Editorial inference: noise augmentation may act as a regularizer that stops the backbone from over-trusting its own previous outputs, a mechanism that could apply to any conditional generative model whose inputs are model-generated."],"forward_implications":["Causal GPT-style decoding on continuous embeddings becomes viable without masking or bidirectional attention, so key-value caching and other decoding optimizations can be used.","The augmentation transfers to other continuous autoregressive formulations: adding it to GIVT with 32 modes lowers its FAD from 0.865 to 0.514 and its FADacc from 0.931 to 0.511.","Because FADacc does not exceed FAD in CAM, the model does not show the usual quality degradation over the second 10 seconds of generation.","The inference-time noise level $k_{\\text{inf}}$ acts as a tunable robustness knob, and at $k_{\\text{inf}} = 0.02$ the long-sequence score actually improves relative to the first 10 seconds.","Purely autoregressive continuous generation can be efficient enough for real-time interactive audio applications, since the sampler is a shallow MLP and denoising cost is small relative to the backbone's forward pass."],"supporting_citations":[{"why":"Defines the autoregressive-continuous baseline (MAR) that CAM improves upon and whose causal configuration fails without masking.","marker":"[5]"},{"why":"Provides the GIVT baseline with Gaussian mixture modeling, which is also tested with the proposed noise augmentation.","marker":"[6]"},{"why":"Supplies the Rectified Flow objective used to train the sampler.","marker":"[16]"},{"why":"Provides the lognormal distribution for the noise level $\\sigma_t$ used in training.","marker":"[17]"},{"why":"The audio autoencoder whose continuous latent embeddings are the sequence elements the model generates.","marker":"[18]"},{"why":"Defines the Fréchet Audio Distance metric used to evaluate generation quality.","marker":"[21]"},{"why":"The CLAP embedding model used to compute FAD, chosen for its stronger correlation with perceived quality.","marker":"[22]"}],"fun_headline_variants":["Noise injection stops error build-up in autoregressive audio","Train with noise to keep autoregressive audio quality over long clips","Random noise in training fixes autoregressive drift in audio synthesis","Noise-augmented CAMs avoid error accumulation in long audio gen"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the assumption that inference errors look like convex combinations of clean embeddings and isotropic Gaussian noise with uniform mixing levels, which the paper does not directly validate.","fun_headline_variants_meta":{"raw":{"variants":["Noise injection stops error build-up in autoregressive audio","Train with noise to keep autoregressive audio quality over long clips","Random noise in training fixes autoregressive drift in audio synthesis","Noise-augmented CAMs avoid error accumulation in long audio gen"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1146,"prompt_tokens":879,"completion_tokens":267,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":196}},"tokens_in":495,"tokens_out":267,"duration_ms":3597,"temperature":1.0,"reasoning_tokens":196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:11:22.029145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the actual prediction errors $e_t = \\hat{x}_t - x_t$ from a trained CAM on long generations and compare their distribution and temporal correlation with the assumed model $k_t \\varepsilon + (1-k_t)x_t$, $k_t \\sim U(0,1)$; if the empirical errors are correlated across timesteps or concentrated at certain mixing levels, training with the uniform schedule should underperform an alternative that uses the measured error distribution.","supporting_citations":[{"cited_title":"Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms","cited_arxiv_id":null,"evidence_quote":"Defines the Fréchet Audio Distance metric used to evaluate generation quality."},{"cited_title":"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation","cited_arxiv_id":null,"evidence_quote":"The CLAP embedding model used to compute FAD, chosen for its stronger correlation with perceived quality."}],"review_version":1}