{"id":"aad6fb7d-64da-4797-82d4-623f2aa81ff0","arxiv_id":"2607.23811","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A streaming encoder plus temporal and fully shared DiT-conditioned depth decoders converts semantic audio tokens to RVQ with constant memory and ~16× real-time on-device synthesis.","lead":"Apple describes a memory-efficient on-device audio detokenizer that turns foundation-model semantic tokens into high-fidelity speech with constant ~21 MB runtime memory. It is the synthesizer behind Siri Expressive Voices and reports faster-than-real-time AMX performance plus MOS gains over prior on-device TTS.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The headline \"constant 21 MB memory at high quality\" is never demonstrated on a single model: memory/latency are measured on the compressed production config (Table 1), while all quality metrics and ablations use the different research config (Table 4, §3.1), and MOS is a full-system comparison that","rationale":"The reader's weakest_assumption identifies the same load-bearing gap: the research/production configuration split and full-system MOS attribution. I confirm this is the right target and sharpen it: the paper's central claim is a conjunction (memory + speed + quality), and the two halves are empirically established on different models, with the production config differing along exactly the quality-determining axes (RVQ levels, bitrate, frame rate). This is a correctness-risk issue for the claim as stated, not a circularity or soundness failure — the architecture itself is coherent, the ablations are reasonable for an industrial report, and the fixed-KV-cache ablation showing zero eval-loss cost is genuine internal support. But since the gap is unverifiable from the text (proprietary data, no production-config quality numbers, no detokenizer-isolated listening test), the reader's CONDITIONAL/MODERATE already prices it correctly. I do not recommend REJECT: the engineering contribution (fully shared depth decoder via DiT stage conditioning, constant-memory streaming) is documented and partially ablated, and nothing in the text is internally inconsistent. I do not recommend ACCEPT: the specific conjunction in the abstract (\"21 MB... enabling high-fidelity streaming\") asserts for the shipped model a quality property measured only on a different configuration. UNCHANGED is the honest recommendation, with the proposed test being the single measurement that would convert this into an accept-shaped claim.","tokens_in":11324,"tokens_out":1453,"duration_ms":59996,"concrete_test":"Evaluate the exact shipped production detokenizer (compressed layer allocation, production RVQ level count, frame rate, bitrate) on the Table 4 battery — UTMOS, SI-SNR, and within-speaker ABX on LibriHeavy — at its production bitrate, and run one isolated MOS A/B holding the AFM front-end fixed while swapping prior vs. new detokenizer. If production-config UTMOS drops below ~3.9 or SI-SNR falls toward the GAN baseline (~5.5), the quality-parity half of the headline claim fails for the deployed model; if the isolated A/B retains most of the +0.28, attribution is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is a conjunction: constant small memory AND quality parity AND MOS gain. But §3.1 states the production model \"differs in several hyperparameters (e.g., layer allocation, number of RVQ levels, frame rate, and bitrate)\" from the research configuration, and Table 4's note confirms \"All entries use the research configuration... the shipped production configuration differs.\" So UTMOS 3.97 / SI-SNR 9.47 and ABX 5.3% are measured on a model that is not the one running at 21 MB / 16x RT on AMX. Compression along exactly the axes that govern quality (RVQ levels, bitrate, frame rate) is precisely where fidelity is lost in codec models; no table shows production-config quality, and no table shows research-config on-device memory. The conjunction is therefore established only by transitivity across two undocumented configs. Second, the +0.28/+0.42 MOS (§3.8) compares the full AFM 3 Core Advanced system (new 1B-activation foundation model + new detokenizer) against the prior TTS — with no isolated detokenizer A/B, the gain is unattributable to the architecture that is the paper's contribution; it could come entirely from the better semantic-token front-end. Note the constant-memory scaling itself is nearly tautological given a fixed-window KV cache (Table 2's flat 1.13 GB follows from construction), so the genuinely empirical content of the claim is quality under that constraint — which is exactly the part not measured on the shipped model. The ablation (Table 3, fixed KV cache costs 0.00 eval loss) is real supporting evidence, but only for the research config.","agreement_with_reader":"agree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper presents the detokenizer architecture behind Siri Expressive Voices: a three-module design (streaming encoder, temporal decoder, depth decoder) that converts semantic audio tokens from the AFM 3 Core Advanced foundation model into RVQ tokens. Its claimed contributions are (i) a single fully shared depth decoder that generates all RVQ levels autoregressively, identifying each level only via DiT-style stage conditioning (RoPE of the codebook index), eliminating Moshi's per-level input/output projections; (ii) causal sliding-window attention with fixed-window KV caching, giving constant runtime memory independent of sequence length; and (iii) on-device deployment on the Apple Matrix Coprocessor at ~10 ms per generation step (~16x real time), ~21 MB peak runtime memory, and 329 MB assets. Empirical support includes a component ablation (Table 3), training-curve studies of conditioning, lookahead, window size, and layer ratios (Figs. 2–4), codec-quality metrics on the research configuration (Table 4: UTMOS 3.97, SI-SNR 9.47), an ABX phonetic discriminability check (5.3%), a GPU scaling comparison (Table 2), and a full-system MOS evaluation (+0.28 overall, +0.42 conversational) against the prior production TTS.","tokens_in":11726,"tokens_out":3961,"duration_ms":83406,"significance":"If the claims hold, this is a practically significant contribution: a production-deployed, streaming semantic-to-RVQ detokenizer that runs at ~16x real time with ~21 MB peak runtime memory on the AMX, enabling on-device expressive TTS alongside a sparsely activated foundation model. Strengths worth crediting: a clear architectural delta over Moshi (fully shared depth decoder with DiT stage conditioning, eliminating per-level projections); a controlled, honestly labeled cross-architecture scaling comparison on a common GPU (Table 2); a reasonably complete component ablation (Table 3) plus conditioning, lookahead, window, and layer-ratio sweeps with training curves; and unusually candid disclosure that the research and production configurations differ. That said, the constant-memory property follows largely from construction (fixed-window KV cache), so the genuinely empirical content — quality under the constraint, on the shipped model — is precisely the part not yet demonstrated, which caps the current evidentiary significance.","major_comments":[{"comment":"The headline claim is a conjunction — constant ~21 MB memory at 16x real time AND synthesis fidelity — but no single model is shown to satisfy both halves. §3.1 states the shipped model 'differs in several hyperparameters (e.g., layer allocation, number of RVQ levels, frame rate, and bitrate)' and that 'unless otherwise stated, the numbers reported refer to the research configuration'; Table 1 (memory/latency) is the production deployment, while Table 4's own note confirms all quality entries (UTMOS 3.97, SI-SNR 9.47) and the 5.3% ABX rate use the research configuration. The production model is compressed along exactly the axes that govern codec fidelity (RVQ levels, bitrate, frame rate). The abstract's claim that the deployed architecture 'maintains synthesis fidelity' is therefore established only by transitivity across two undocumented configs. Please report production-config UTMOS/SI","section":"§3.1, Table 1 vs. §3.7, Table 4"},{"comment":"The +0.28 overall / +0.42 conversational MOS compares the full AFM 3 Core Advanced system — a new 1B-activation foundation model plus the new detokenizer — against the prior TTS. There is no isolated detokenizer A/B, so the gain cannot be attributed to the architecture that is the paper's contribution; it may derive entirely from the improved semantic-token front-end. Either add a same-front-end detokenizer comparison or reframe the MOS result as a system-level outcome. Separately, no listener counts, confidence intervals, or significance testing are given, so the reliability of the 0.1-MOS-resolution claims cannot be judged.","section":"§3.8 (Production Human Evaluation)"},{"comment":"The text states VisQOL, MOSNet, and SI-SNR are used to measure acoustic quality, but VisQOL and MOSNet results never appear anywhere in the paper. Likewise, the text says ABX error rates are compared 'with three different baseline models' (Mimi decoder, internal transformer decoder, internal AR-GAN), yet only the proposed model's 5.3% is reported. Finally, Table 4 compares 'Ours' at 1.5 kbps / 25 Hz against Moshi/Mimi at 1.1 kbps / 12.5 Hz — not bitrate- or frame-rate-matched — which weakens the 'matching or beating compared codecs' claim in the abstract. Please report the promised numbers and add a bitrate-matched comparison.","section":"§3.7 (Audio Quality Evaluation) and Table 4"},{"comment":"(a) The table note and §3.1 text claim 'our model holds generation time, memory, and RTF constant with sequence length,' but the table itself shows generation time scaling 2.40 → 9.64 → 38.59 s — linear, with RTF constant at 0.12. The sentence is contradicted by the data. (b) The flat 1.13 GB memory follows from construction of the fixed-window KV cache, so the scaling table's empirical content is the speed/quality behavior, not the memory constancy. (c) Contribution (4) and the abstract claim superior speed 'including Moshi,' but Moshi does not appear in Table 2, and at 512 tokens the transformer-decoder baseline is faster than the proposed model (RTF 0.04 vs. 0.12) — the speed advantage holds only at long sequence lengths. Please correct the note, add Moshi or drop it from the speed claim, and scope the claim to the long-sequence regime.","section":"§3.1, Table 2 and surrounding text"}],"minor_comments":[{"comment":"All rows are evaluated by token-level eval loss only; since Table 4 shows loss and perceptual quality need not track each other, reporting UTMOS or SI-SNR for at least the key ablations (lookahead, unified depth decoder) would strengthen the claim that these components 'validate' audio quality, not just token prediction.","section":"Table 3 (Ablation Study)"},{"comment":"'Audio cleanliness' (fraction free of tremor, slurs, glitches) is introduced without a definition, annotation protocol, or rater information; 'Navigation reaching 100% clean' is anecdotal as stated.","section":"§3.8"},{"comment":"No table specifies the research configuration itself (parameter counts per module, RVQ levels K, codebook size, window W used in the final model). Since the production config is undisclosed, the research config is the only reproducible artifact and should be fully specified.","section":"§2 / §3"},{"comment":"Two empirical claims are asserted without numbers: removing the RVQ autoregressive loop 'leads to a severe drop in generated RVQ token accuracy' (§2), and bidirectional encoding gives 'marginal accuracy improvements' (§3.3). Please quantify or cite a figure.","section":"§2 (temporal decoder paragraph)"},{"comment":"Eqs. (2.1)–(2.7): the role of RoPE in Eq. (2.3) — applying rotary encoding of the codebook index to the temporal-decoder output as the sole level identifier — deserves one more sentence, since it is the mechanism distinguishing the model from Moshi. Also clarify whether qt,k in Eq. (2.5) is the codebook embedding of the sampled token rt,k.","section":"Figure 1 and Eqs. (2.1)–(2.7)"},{"comment":"Grammar: 'Inspired from Moshi' → 'Inspired by Moshi'; 'in order to measure' can be simplified. The insets in Figures 2–4 are described in captions but their axes are difficult to read at print size.","section":"§3.7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript doubles as a product announcement (Siri Expressive Voices, AFM 3 Core Advanced branding in the abstract), and the training corpus, research-config hyperparameters, and production-config details are undisclosed, so independent reproduction is impossible. The authors are at least transparent that the research and production configurations differ — the gap is disclosed, not hidden — but the abstract's combined efficiency-plus-quality claim still leans on the undisclosed configuration. Concurrent work (Fish Audio S2) is acknowledged in the text. Whether the systems/product framing fits the journal's scope is the editor's call; technically the architecture is a credible, incremental refinement of Moshi's depth transformer rather than a new paradigm."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that this is a solid industrial systems paper on semantic-to-RVQ detokenization under a hard AMX budget. The distinctive move relative to Moshi is a single depth body whose parameters are fully shared across RVQ levels and identified only by DiT-style stage/RoPE conditioning—no per-level projections. Fixed-window causal attention then makes runtime memory independent of length. That combination is practical and clearly motivated for co-running with a sparse on-device foundation model.\n\nWhat they do well: equations 2.1–2.7 are readable, the component ablations (lookahead, unified depth, DiT vs cross-attn, output norm, fixed KV) are systematic, and Table 2 shows the expected flat memory/time scaling against transformer and AR-GAN baselines. Codec numbers on the research config (UTMOS 3.97, SI-SNR 9.47, ABX 5.3%) look competitive with Mimi and their internal baselines. They also flag concurrent Fish Audio S2 and are explicit that production differs in layers, K, frame rate, and bitrate.\n\nThe soft spot the stress-test flags is real and proportionate: Table 1’s ~21 MB / 16× RT is the compressed production model; Table 4 and the ablations are the research model; the +0.28/+0.42 MOS is the full AFM 3 stack versus prior TTS, not an isolated detokenizer A/B. So the headline conjunction (tiny constant memory AND quality parity AND MOS gain) is established by transitivity across undocumented configs. Constant memory itself is nearly by construction once you fix the KV window; the empirical claim that matters is quality under that constraint, and that is not measured on the shipped footprint. Proprietary data and no code are expected for this venue but limit external check.\n\nNone of that makes the architecture unserious. For people building edge TTS/codecs or hierarchical RVQ decoders, the shared-depth + fixed-window pattern and the ablation curves are useful. I would send it to referees; they will ask for a production-config quality table or a clearer attribution split, which is fixable. Worth engaging if you work on on-device audio; skip if you only care about open, reproducible science claims.","headline":"Clean on-device systems advance: fully shared DiT-conditioned depth decoder plus fixed-window KV gives real constant-memory detokenization; the quality–memory conjunction is split across two configs.","tokens_in":13008,"tokens_out":574,"would_cite":true,"duration_ms":14476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A three-part detokenizer turns semantic audio tokens into high-fidelity speech at constant ~21 MB memory and ~16× real time on device.","keywords":["Speech Synthesis","Residual Vector Quantization","Diffusion Transformers","High-fidelity Synthesis","On-device detokenizer","Sliding-window attention","Constant memory"],"falsifier":"Run the shipped production detokenizer alone (same bitrate/frame rate as research) on held-out long utterances and check whether UTMOS/SI-SNR and ABX stay at or above the reported research numbers while peak runtime memory remains near 21 MB and independent of length out to several minutes.","tokens_in":12685,"feed_emoji":"🔊","tokens_out":892,"duration_ms":17218,"temperature":0.7,"pith_summary":"This paper presents the on-device audio detokenizer behind Siri Expressive Voices: a system that converts semantic tokens from a foundation model into residual vector quantization (RVQ) codes, then into speech, inside a tight shared memory budget. It splits the job into a streaming encoder, a temporal decoder, and one reusable depth decoder. The depth decoder generates every RVQ level with fully shared weights, guided only by Diffusion Transformer–style stage conditioning, instead of separate per-level decoders or projections. Causal sliding-window attention with a fixed-window key-value cache keeps runtime memory flat no matter how long the utterance is. On the Apple Matrix Coprocessor the design runs about 10 ms per step (~16× real time) at ~21 MB peak runtime memory, supports continuous 20–320 second streams, and, inside the full on-device stack, lifts Mean Opinion Score by +0.28 overall and +0.42 on conversational speech versus the prior production TTS.","feed_headline":"On-device speech detokenizer runs 16× real time at 21 MB","feed_subtitle":"One shared depth decoder and fixed-window cache keep memory flat from 20 to 320 seconds of audio","key_machinery":"The three-component detokenizer—streaming encoder, temporal decoder, and one reusable depth decoder with DiT-style stage conditioning (rotary encoding of the codebook index) plus fixed-window KV caching—carries the argument by making depth generation parameter-shared and memory independent of sequence length.","core_discovery":"Decoupling temporal processing from depth (RVQ-level) processing, and identifying each RVQ level only through DiT-style stage conditioning on a single fully shared depth decoder, yields constant-memory, faster-than-real-time semantic-to-RVQ conversion that preserves synthesis quality under on-device constraints.","pith_inferences":["The same temporal–depth split and stage-conditioned shared decoder could transfer to other multi-codebook codecs beyond speech, wherever memory must stay flat under streaming.","If mean RVQ embedding feedback is the main temporal glue, replacing it with a learned temporal summary might further cut depth iterations without raising memory.","Constant-memory detokenizers lower the barrier to always-on multimodal assistants that keep audio generation resident while routing other experts in and out of DRAM."],"forward_implications":["Streaming expressive speech can run continuously on device beside a sparsely activated foundation model without memory growing with utterance length.","Per-level decoder stacks and per-codebook input/output projections become unnecessary once stage conditioning fully identifies RVQ depth.","Fixed-window KV caching makes 20–320 second synthesis practical under a shared DRAM budget of tens of megabytes.","Full-system MOS gains of about +0.3 overall and +0.4 on conversational speech become achievable at a 1B-parameter activation size on device."],"fun_headline_variants":["Shared depth decoder holds speech synthesis to 21 MB flat","Decoupled DiT depth path runs detokenizer 16× real time","One stage-conditioned decoder replaces per-level RVQ models","Fixed-window cache keeps memory flat across 320s audio","Temporal-depth split yields constant-memory on-device speech"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That quality and speed measured on the research model still hold for the separately compressed production detokenizer, and that fixed-window attention plus mean RVQ feedback keep enough long-range context for high-fidelity streaming.","fun_headline_variants_meta":{"raw":{"variants":["Shared depth decoder holds speech synthesis to 21 MB flat","Decoupled DiT depth path runs detokenizer 16× real time","One stage-conditioned decoder replaces per-level RVQ models","Fixed-window cache keeps memory flat across 320s audio","Temporal-depth split yields constant-memory on-device speech"]},"model":"grok-4.5","effort":"low","cost_usd":0.004833,"raw_usage":{"total_tokens":1478,"prompt_tokens":900,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":48328000,"prompt_tokens_details":{"text_tokens":900,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":506,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":900,"tokens_out":72,"duration_ms":9368,"temperature":1.0,"reasoning_tokens":506,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T11:45:24.945881+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the shipped production detokenizer alone (same bitrate/frame rate as research) on held-out long utterances and check whether UTMOS/SI-SNR and ABX stay at or above the reported research numbers while peak runtime memory remains near 21 MB and independent of length out to several minutes.","supporting_citations":[],"review_version":1}