{"id":"f72602e1-de6e-4185-9122-40b5735f89c1","arxiv_id":"2506.14761","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An autoregressive U-Net trained directly on bytes, with pooled word and multi-word stages, matches and often beats a BPE-tokenized Transformer at similar compute.","lead":"This paper presents a byte-level language model that groups raw text into words and multi-word chunks by itself, without a fixed tokenizer, using a multi-scale U-Net. Under similar training compute it matches or beats a standard token-based Transformer on several benchmarks, with extra gains on low-resource languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BPE baseline at the 370B-token scale uses LR=0.001, 3x below the paper's own fitted formula (0.003) for the reported compute, while AU-Net uses its formula; the headline match may be against an undertuned baseline.","rationale":"The reader correctly identified compute-fairness as the load-bearing axis, but the specific weak point I find is not the FLOP formula itself. The paper's own hyperparameter formulas in Section 2.3 are inconsistent with Table 11 for the BPE baseline at the 370B-token scale: the formula predicts LR about 0.0030, while the table reports 0.001, a 3x gap. AU-Net runs at the same scale use values within about 15% of their formula. This asymmetry means the BPE baseline may have been undertuned, which would directly undermine the headline 'AU-Net matches or outperforms BPE at equal compute' claim. The warmup discrepancy (Table 6 says 10%, Table 11 shows 0.6% for the 310k-step run) further indicates that the hyperparameter reporting is not reliable enough to trust without an explicit check. I do not reject the paper, because the discrepancy could be a table typo or a deliberate but undocumented deviation for the over-trained regime; the proposed re-run settles that. FLOP accounting remains a secondary concern, but it is not the most direct threat, especially since the reported AU-Net FLOPs are consistent with observed H100 throughput and any error in that direction appears conservative. Verdict stays CONDITIONAL, matching the reader's conditional assessment.","tokens_in":23695,"tokens_out":31595,"duration_ms":312607,"concrete_test":"Re-run the BPE Transformer at 370B tokens (same model, data, compute, and released code) with peak LR=0.0030, the value of Section 2.3's formula at C=3.6e21, and, as a control, LR=0.001 as in Table 11, with all other settings from Tables 10/11 fixed. Compare Hellaswag and MMLU, the two flagship gains. If LR=0.0030 improves either by more than about 1-2 points, the AU-Net headline advantage is likely an undertuned-baseline artifact; if scores are equal or worse, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AU-Net matches or beats BPE under equal compute depends on the BPE baseline being well tuned at the same budget. Section 2.3 reports LR_BPE(C)=19.3*C^-0.177 and says these formulas are used to extrapolate optimal hyperparameters for any compute budget. For the decisive 370B-token run, Table 10 ('Transformer 1B dm8 BPE') lists Total FLOPs 3.6e21; the formula gives LR=19.3*(3.6e21)^-0.177, approximately 0.0030. Table 11, however, reports LR=0.001 for exactly this run, a 3x deviation. The AU-Net dm8 runs, by contrast, are consistent with their formula: AU-Net 2 uses 0.00094 vs formula 0.00108; AU-Net 3 uses 0.0011 vs 0.00101; AU-Net 4 uses 0.0011 vs 0.00096. If the BPE baseline was trained at a learning rate far below its own extrapolated optimum, the headline gains (Hellaswag 73.7 vs 70.2, MMLU 31.7 vs 27.0) could be an artifact of an undertuned baseline rather than a real architectural advantage. This concern is distinct from FLOP accounting; even perfect FLOP counts cannot fix a hyperparameter mismatch. The inconsistency is corroborated by Table 6's claim of 10% warmup, which is contradicted by Table 11's 2000-step warmup for a 310k-step run (0.6%). The check below settles whether the concern lands.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Autoregressive U-Net (AU-Net), a byte-level language model that replaces a fixed tokenizer with learned, hierarchical pooling at word, word-pair, and four-word boundaries. The architecture processes raw bytes in a contracting path, applies global attention (with window attention at the byte stage), and reconstructs the sequence via position-specific upsampling and skip connections. The authors report scaling laws for batch size and learning rate, and present pretraining experiments at 1B and 8B scale on DCLM, comparing AU-Net against BPE-tokenized Transformers, byte-level Transformers, and Mamba. The central claim is that, under controlled compute budgets, shallow AU-Nets match or outperform strong BPE baselines, deeper hierarchies show promising scaling trends, and byte-level operation gives advantages on character-level tasks and low-resource-language translation.","tokens_in":23971,"tokens_out":6062,"duration_ms":55525,"significance":"If the central parity claim holds, this is a meaningful step toward tokenizer-free language modeling: the model learns token embeddings dynamically, supports multi-scale prediction, and demonstrates transfer to languages and tasks where fixed tokenizers are brittle. The paper ships code in Meta Lingua, reports bootstrap confidence intervals, and includes extensive ablations on pooling, upsampling, and layer allocation, which are concrete strengths. However, the headline comparison rests on the fairness of the BPE baseline's hyperparameters and on the accuracy of the FLOP accounting; both need to be secured before the parity claim can be accepted.","major_comments":[{"comment":"The BPE baseline at the decisive 370B-token budget is trained with LR=0.001 (Table 11, 'Transformer 1B dm8 BPE'), but the paper's own formula LR_BPE(C)=19.3*C^-0.177 (Section 2.3) gives approximately 0.003 for C=3.6e21 (Table 10). This is a 3x deviation, and it contradicts the stated protocol that the formulas are used to extrapolate optimal hyperparameters for any compute budget (Section 2.3, Section 3.1). Because Table 1 reports AU-Net 3 and AU-Net 4 beating this BPE baseline on Hellaswag, MMLU, and GSM8k, the headline gains may be an artifact of an undertuned baseline rather than an architectural advantage. Please rerun the BPE baseline at the formula learning rate and report the results, or provide a direct ablation showing that LR=0.001 is optimal for this budget.","section":"Section 2.3, Table 10, Table 11"},{"comment":"Table 6 states that warmup spans 10% of total training steps, but Table 11 reports 2000 warmup steps for the 310k-step BPE run (about 0.6%) and for scaling runs with 14k-85k steps (up to 14%). These inconsistencies make the training setup irreproducible and undermine the paper's claim of 'carefully tuning' hyperparameters. Please correct the tables or clarify the actual warmup schedule used for each run.","section":"Section 3.1, Table 6, Table 11"},{"comment":"Contribution C3 claims the comparison is made 'under identical pre-training budgets,' but Table 1 lists 3e21, 4e21, and 5e21 FLOP for AU-Net versus 4e21 for the BPE baseline. The budgets differ by up to 25%, and the text itself elsewhere describes the comparison as being 'within 25%.' Please correct the overstatement and state explicitly how the compute differences are handled when drawing conclusions from Table 1 and Figure 3.","section":"Section 1 (Contribution C3), Table 1"},{"comment":"The data-to-model ratio used for the scaling-law comparison is ambiguous. Section 3.2 ends with 'We focus on AU-Net 2 and AU-Net 3, using a data-to-model ratio of 2,' while Section 3.3 says the baseline is trained 'with a data-to-model ratio of 10.' This ambiguity directly affects the interpretation of Figure 3. Please specify which data-to-model ratio was used for each architecture and ensure the comparison is made at equal ratios.","section":"Section 3.2, Section 3.3"},{"comment":"The Total FLOPs, FLOPs/Token, and Tokenizer entries are listed as 'nan' for the byte-level Transformer and Mamba baselines, despite Section 3.1 claiming that all baselines were trained 'with the same amount of data or compute.' The FLOP comparison for these baselines cannot be audited. Please provide the missing values or explain why they are unavailable.","section":"Table 10"}],"minor_comments":[{"comment":"The FLOP formula for AU-Net in Section 2.3 assumes full attention at every stage, but Section 2.1 states that the byte-level stage restricts attention to a window. Please clarify whether the reported FLOPs overestimate the actual compute of AU-Net due to window attention; if so, the comparison is conservative, but this should be stated explicitly.","section":"Section 2.1, Section 2.3"},{"comment":"The caption says the table reports 'compute budget and empirical training speeds in bytes/sec,' but the column is labeled 'Flops.' Please unify the terminology so readers can interpret the column correctly.","section":"Table 2 caption"},{"comment":"The abstract says shallow hierarchies 'tie strong BPE baselines,' while Table 1 shows AU-Net 3 and AU-Net 4 outperforming the baseline on several tasks. After the baseline-tuning issue is resolved, please align the wording with the quantitative results.","section":"Abstract, Section 1"},{"comment":"The claim of an 'Infinite vocab size' is overstated, since the model still has fixed hidden dimensions and a finite sequence length; consider rephrasing to 'unbounded vocabulary' to avoid confusion.","section":"Section 1 (Contribution C2)"},{"comment":"There is a typo in the table description: 'compite budget' should be 'compute budget.'","section":"Appendix D, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's own tables contain a concrete inconsistency that undermines the central parity claim: the BPE baseline at the 370B-token scale uses a learning rate three times below the value given by the paper's own scaling-law formula. This is a fixable but load-bearing issue. If the authors can rerun the baseline at the correct learning rate (or justify the lower value), the paper could make a strong contribution. I would not recommend rejection, because the architecture is novel and the experimental apparatus is largely sound, but the reported head-to-head results cannot be accepted as-is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know? This is a genuine, mostly careful empirical paper: byte-level autoregressive U-Net with up to four pooling stages, global attention per stage, skip connections, and position-specific upsampling. The combination is new relative to MegaByte, BLT, and Nawrot et al. It ships code, detailed config tables, and sensible ablations. The byte-level advantages on CUTE character manipulation and FLORES low-resource translation look real, and the authors are honest about the space-delimited-script limitation and FSDP overhead. The non-circular comparisons against external benchmarks are a plus.\n\nThe soft spot, though, is load-bearing. The flagship Table 1/Table 2 comparison at 370B tokens uses a BPE baseline with LR=0.001 (Table 11), while the paper's own formula LR_BPE(C)=19.3*C^-0.177 for C=3.6e21 gives ~0.003. The AU-Net runs use LRs consistent with their formula. If that BPE run is undertuned, the headline gains (Hellaswag 73.7 vs 70.2, MMLU 31.7 vs 27.0) may be an artifact. This is not a minor footnote: it is the central parity claim. The paper also says warmup is 10% of steps in Table 6, but the same BPE run has 2000 steps warmup out of 310k (0.6%), another internal inconsistency.\n\nOther soft spots: the FLOP entries for byte-level Transformer and Mamba baselines are nan in Table 10, so the 'matched compute' claim for those baselines is unauditable; the byte-to-token conversion relies on a single measured compression factor k=4.56; and the claim that hierarchical models 'consistently match or outperform' overstates the TQA, NQ, and GSM8k results, where AU-Net loses or ties. The scaling-law extrapolations come from miniaturized runs, which is standard practice, but they deserve scrutiny.\n\nOn balance: the architecture and much of the empirical work are solid, and the 60B-token and 8B-scale results give some support for the parity claim independent of the 370B-token comparison. But the authors need to either re-run or re-report the BPE baseline at its formula LR, or explain the deviation, and clean up the warmup inconsistency, before the headline claim is trustworthy. This deserves peer review—an editor should send it out—but my own verdict would be conditional on fixing the baseline tuning and FLOP accounting.","headline":"A credible tokenizer-free architecture study whose headline 'matches or beats BPE' claim is undercut by a 3x learning-rate deviation in the BPE baseline at the flagship 370B-token scale.","tokens_in":24658,"tokens_out":3153,"would_cite":true,"duration_ms":29590,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A byte-level autoregressive U-Net matches BPE-tokenized Transformers under matched pretraining compute.","keywords":["byte-level language modeling","autoregressive U-Net","tokenizer-free","hierarchical pooling","multi-scale prediction","scaling laws","low-resource language transfer","character-level tasks"],"falsifier":"Measure the true training FLOPs with a hardware profiler for AU-Net 4 and the BPE baseline at the 1B/370B-token configuration; if AU-Net 4's real cost exceeds the reported 5e21 by an amount that closes the gap with BPE's 4e21, or if the re-measured byte-level Transformer and Mamba FLOPs, currently listed as nan, come out far above their token-based counterparts, the headline parity result fails. A simpler test is to retrain the byte-level Transformer baseline on the same 370B-byte budget and check whether it also narrows the Hellaswag and MMLU gap, which would attribute the gain to more bytes rather than to the hierarchy.","tokens_in":23356,"feed_emoji":"🧠","tokens_out":6496,"duration_ms":53972,"temperature":0.7,"pith_summary":"Tokenization freezes the granularity of language modeling before training even begins. This paper proposes an autoregressive U-Net that reads raw bytes and pools them into words, word pairs, and four-word chunks at successive stages, so the model learns its own token representations and predicts further ahead at deeper stages. The central claim is that, under carefully matched pretraining compute, these hierarchical byte-level models match or outperform Transformer baselines that use BPE tokenization, with deeper hierarchies showing a stronger advantage on several reasoning tasks. Because tokenization lives inside the model, the same system also handles character-level tasks and carries knowledge to low-resource languages. If the claim holds, tokenizer-free architectures become a viable alternative to standard tokenization and its fixed vocabularies.","feed_headline":"Tokenizer-free U-Net matches BPE LLMs on equal compute","feed_subtitle":"Deep byte-level hierarchies beat BPE on reasoning and low-resource language translation.","key_machinery":"The central object is the autoregressive U-Net (AU-Net), a contracting-expanding network with adaptive pooling and multi-linear upsampling. The contracting path compresses the byte sequence at each stage by keeping only vectors at positions given by a splitting function (word boundaries, then two-word and four-word chunks), while the expanding path restores sequence length by duplicating coarse vectors and applying position-specific linear layers, combined with skip connections from the contracting path. The workhorse identity for the compute comparison is the FLOP-per-byte formula $F_{\\text{model/byte}} = \\sum_{i=1}^{L} F_i^{\\text{model/byte}} / k_i$, where $k_i$ is the contraction factor at stage $i$, together with the conversion $\\gamma_{\\text{byte}} = k^2 \\gamma_{\\text{token}}$ using the measured byte-to-token compression $k \\approx 4.56$ on DCLM. Deeper stages predict further ahead, so they focus on broad semantic patterns while earlier stages refine local detail.","core_discovery":"The paper claims that an autoregressive U-Net operating on raw bytes can be made compute-competitive with BPE-tokenized Transformers. Specifically, at a 1B-parameter-equivalent scale trained on 370B tokens, the four-stage AU-Net scores 73.7 versus 70.2 on Hellaswag, 31.7 versus 27.0 on MMLU, and 5.3 versus 4.4 on GSM8k against the BPE baseline, at a comparable total FLOP budget (5e21 versus 4e21). The load-bearing assertion is that a hierarchical model that contracts sequences via pooling at word boundaries and expands them via position-specific upsampling can match a strong tokenized baseline at the same compute, while also generalizing better to low-resource languages and character-level tasks. The deeper stages anticipate several words ahead, functioning like multi-token prediction without auxiliary losses.","pith_inferences":["If the FLOP accounting is correct, then the fixed overhead of BPE vocabularies, such as a 525M-parameter embedding in a 1B model, is replaceable by compute spent on hierarchical attention, inverting the usual assumption that embeddings are cheap while attention is expensive.","The $k^2$ scaling of the data-to-model ratio means byte-level models are effectively over-trained relative to token-based models on the same raw data; the measured advantage on low-resource languages could come from this over-training plus shared orthography rather than from hierarchy per se, a distinction this paper does not test.","Because the splitting function is a regular expression over spaces, extending AU-Net to non-space-delimited scripts will likely require learning the splitting function; the paper's own Chinese MMLU drop suggests this as the next concrete extension.","The reported 'nan' FLOP entries for the byte-level Transformer and Mamba baselines mean the parity claim currently rests on the paper's own accounting; an independent profiler-based comparison would either confirm or overturn the result."],"forward_implications":["Tokenizer-free language models can reach parity with BPE-tokenized Transformers at matched pretraining compute, making the embedding table and fixed vocabulary optional.","Deeper hierarchies act as implicit multi-token prediction: the deepest stage anticipates several words ahead, which may substitute for explicit auxiliary prediction targets.","The same model can switch between raw-byte and word-level granularity, enabling character-level manipulation tasks and better cross-lingual transfer to low-resource or unseen languages without retraining a tokenizer.","Scaling laws for AU-Net require different batch-size and learning-rate formulas ($BSZ_{\\text{AU-Net}}(C) = 0.66\\,C^{0.321}$ and $LR_{\\text{AU-Net}}(C) = 6.6\\times C^{-0.176}$), which are reported to give smooth optimization and predictable downstream gains.","Fair comparisons between byte-based and token-based training must rescale the data-to-model ratio by the square of the compression factor, $k^2$, rather than comparing raw byte and token counts."],"supporting_citations":[{"why":"Supplies the U-Net architecture pattern of contracting and expanding paths with skip connections that AU-Net adapts.","marker":"Ronneberger et al., 2015"},{"why":"Defines the prior hierarchical transformer U-Net that AU-Net extends from fixed pooling to adaptive pooling and from perplexity evaluations to downstream tasks.","marker":"Nawrot et al., 2022"},{"why":"Provides the multi-token prediction idea that AU-Net's deeper-stage long-horizon prediction mirrors without auxiliary losses.","marker":"Gloeckle et al., 2024"},{"why":"BLT is the byte-level latent transformer baseline and an entropy-splitting precedent that AU-Net contrasts with on architecture and pooling design.","marker":"Pagnoni et al., 2024"},{"why":"A hierarchical autoregressive transformer that combines byte- and word-level processing, serving as a direct comparison point for adaptive multi-stage pooling.","marker":"Neitemeier et al., 2025"},{"why":"Establishes the scaling-law protocol for sweeping learning rate and batch size on small models to predict optimal hyperparameters for larger ones, which AU-Net adapts to byte-level training.","marker":"Bi et al., 2024"},{"why":"Establishes the data-to-model ratio gamma as a key quantity for smooth scaling laws, which the paper uses to control compute budgets.","marker":"Gadre et al., 2024"},{"why":"Provides the DCLM pretraining corpus and the sequence-packing with full-attention training prescription used in all experiments.","marker":"Li et al., 2024"},{"why":"The LLaMa 3 tokenizer defines the BPE baseline and supplies the measured compression factor k approximately 4.56 on DCLM.","marker":"Grattafiori et al., 2024"},{"why":"Mamba is the byte-level state-space baseline that AU-Net is compared against on equal data and compute budgets.","marker":"Gu and Dao, 2024"}],"fun_headline_variants":["Byte-level U-Net ties BPE LLMs on identical compute","Autoregressive U-Net matches tokenized LLMs on bytes","No tokens needed: U-Net rivals BPE at same FLOPs","Deep byte hierarchy U-Net equals BPE LLMs on compute","Byte U-Net matches BPE Transformers on equal flops"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the FLOP-accounting identity for AU-Net, together with the byte-to-token budget conversion using the 4.56 compression factor, measures the same pretraining budget as BPE; if that accounting is biased, the parity claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Byte-level U-Net ties BPE LLMs on identical compute","Autoregressive U-Net matches tokenized LLMs on bytes","No tokens needed: U-Net rivals BPE at same FLOPs","Deep byte hierarchy U-Net equals BPE LLMs on compute","Byte U-Net matches BPE Transformers on equal flops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1451,"prompt_tokens":902,"completion_tokens":549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":456}},"tokens_in":518,"tokens_out":549,"duration_ms":5245,"temperature":1.0,"reasoning_tokens":456,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:48:20.836185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the true training FLOPs with a hardware profiler for AU-Net 4 and the BPE baseline at the 1B/370B-token configuration; if AU-Net 4's real cost exceeds the reported 5e21 by an amount that closes the gap with BPE's 4e21, or if the re-measured byte-level Transformer and Mamba FLOPs, currently listed as nan, come out far above their token-based counterparts, the headline parity result fails. A simpler test is to retrain the byte-level Transformer baseline on the same 370B-byte budget and check whether it also narrows the Hellaswag and MMLU gap, which would attribute the gain to more bytes rather than to the hierarchy.","supporting_citations":[],"review_version":2}