{"id":"30ba77c6-a6c7-477d-9499-4045cc88b25d","arxiv_id":"2608.03599","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper hypothesizes that next-byte and boundary distributions in byte-level LMs can be disentangled, proposes two experiments to test it, but provides no experimental results.","lead":"This position paper argues that byte-level language models can share knowledge exactly while keeping their own chunking behavior, and that these two aspects can be adjusted independently. It proposes two experiments to test this, and calls for a shared byte-level interface to replace per-model tokenizers.","discovery_kind":"unclear","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The disentangling hypothesis rests on unreported preliminary measurements; absent those data, the central claim is unsupported.","rationale":"The reader's rationale centers on the absence of supporting data, and I agree: the central claim is asserted as a hypothesis but is accompanied by unshown 'preliminary measurements' and 'initial results' that are load-bearing for its plausibility. However, the reader's weakest_assumption specifically identifies embedding resettability as the critical assumption, whereas my concern is broader: even granting that byteification preserves capabilities, the disentangling hypothesis itself has no empirical support. The proposed experiments are not results, and the unreported measurements are the only empirical legs. I would keep the REJECT verdict because the paper does not meet the standard of a research result, but the precise weakest point is the missing evidence for the disentangling hypothesis rather than only the resettability prerequisite.","tokens_in":3627,"tokens_out":8719,"duration_ms":92400,"concrete_test":"Reproduce the Section 2 'high-divergence' measurement: obtain or train two byte-level models, one derived from a SentencePiece-tokenized model and one from a BPE-tokenized model, and compute the boundary divergence (1 - F1 between their per-byte boundary predictions) on a fixed held-out corpus. If the claimed high-divergence settings do not appear, or if the numbers differ by more than a small margin from those the authors would report, the paper's empirical motivation collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the hypothesis that the next-byte and boundary-placement distributions can be changed almost independently. The only empirical motivation offered is a set of unreported results: (i) 'our initial results confirm embedding resettability holds on Ukrainian models too' (Section 2), (ii) 'we have already conducted some initial measurements across different domains and model families, and found promising, high-divergence settings' (Section 2), and (iii) a claim that Bolmo's predicted boundaries barely differ from the OLMo tokenizer it was byteified from. None of these measurements are included, and they are not derivable from the cited Bolmo work, which demonstrates byteification but not disentangling. The two experiments in Section 3 are explicitly prospective ('will distill', 'will try'), so the paper contains no experimental evidence for the central claim. If the preliminary measurements are unavailable or unreproducible, the hypothesis has no empirical foundation; it is a bare conjecture. This is a missing-support problem, not a disagreement with consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This preprint argues that byte-level language models have a structural advantage beyond tokenizer robustness: because all byte-level models emit distributions over the same 256-byte alphabet, knowledge transfer between them is exact. The paper's central hypothesis is that the next-byte distribution and the patch-boundary distribution produced by such a model can be disentangled and changed almost independently, so a model could acquire a teacher's capabilities without changing its boundaries, or change its boundaries without losing capabilities. The paper motivates this idea via Bolmo-style byteification, defines boundary divergence as 1−F1, references (without showing) preliminary measurements on Ukrainian models and high-divergence settings, and describes two prospective experiments: (i) next-byte distillation between two Bolmo models while holding boundaries fixed, and (ii) boundary-predictor retraining while self-distilling the language-modeling distribution. It closes with a call for a shared byte-level interface.","tokens_in":3915,"tokens_out":5546,"duration_ms":57259,"significance":"If the hypothesis were established, the paper would identify a genuinely useful design point: a common byte interface would make cross-tokenizer distillation exact and cheap, and would allow tokenizer/boundary choice to be adjusted separately from language-modeling capabilities. The proposed experiments are concrete and falsifiable, and the focus on Bolmo's joint p(b,m) is a good testbed. These are real strengths. As it stands, however, the manuscript contains no experimental evidence; the preliminary measurements it invokes are not included, and the two decisive experiments are only planned. The contribution is therefore currently a plausible position piece rather than a demonstrated result. The missing data are the difference between a supported claim and a conjecture.","major_comments":[{"comment":"The manuscript claims two load-bearing empirical facts but reports none of the measurements: (i) 'our initial results confirm embedding resettability holds on Ukrainian models too' and (ii) 'we have already conducted some initial measurements across different domains and model families, and found promising, high-divergence settings', most clearly between SentencePiece- and BPE-derived models. The boundary-divergence metric is defined as 1−F1 (Section 2), yet no F1 values, model names, domains, dataset sizes, or experimental procedures are given. The second experiment also relies on the claim that a Bolmo model's boundaries 'barely differ' from the OLMo-family tokenizer it was byteified from, again without numbers. These are not optional extras: they are the empirical foundation for the disentangling hypothesis and for the feasibility of the proposed boundary-retraining experiment. The au","section":"Section 2 (Preliminaries), boundary-divergence definition"},{"comment":"The first experiment 'will distill one Bolmo model into another over the language modeling distribution alone, leaving each model's boundaries untouched.' The premise that next-byte distillation leaves the boundary marginal p(b,m) unchanged is the core of the 'disentangling' claim, but the paper provides no argument or evidence for it. The shared byte alphabet makes the target distribution well-defined, but it does not imply independence between the two marginals of the joint output, nor that the internal boundary predictor is unaffected by distillation. The title's 'Disentangling' asserts more than the text establishes. A proof-of-concept result, or at least a formal statement of the independence claim, is needed before the hypothesis can be regarded as supported.","section":"Section 3, first experiment"},{"comment":"The second experiment aims to change boundaries while keeping capabilities, but the paper itself acknowledges a coupling: 'the boundaries decide how bytes are pooled into patches and so reshape the very input the backbone transformer sees' (Section 3). Since the backbone operates on patched inputs, changing the boundary predictor changes the model's input distribution, so it is not obvious how self-distilling the language-modeling distribution can 'keep it fixed' without altering the model's internal representations. The paper does not specify the self-distillation loss, the data, or the agreement procedure between the auxiliary boundary predictor and the decoding patching head. This is a load-bearing gap in the second direction of the hypothesis.","section":"Section 3, second experiment"}],"minor_comments":[{"comment":"The quantitative claim that Bolmo byteifies 'for under a percent of pretraining cost' is not attributed to a specific result in Minixhofer et al. (2025a); either cite the precise number or soften the wording.","section":"Section 3"},{"comment":"The phrase 'the right common alphabet' is left undefined. Since the paper argues against per-model tokenizers, it should state what alternatives to bytes are being considered (e.g., grapheme/CLP) or explicitly defer that question.","section":"Section 3"},{"comment":"The abstract promises 'preliminary measurements of the properties they rest on', but Section 2 contains only qualitative descriptions. The mismatch should be fixed even after adding the data.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a position paper. If the venue's scope requires empirical validation rather than proposals, the absent preliminary data is disqualifying; I would reject. If position papers are in scope, the two planned experiments and the falsifiable framing make the paper salvageable, provided the author supplies the referenced measurements and either weakens the title or reports actual evidence for the disentangling claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a position paper, not a results paper. The author proposes that for byte-level LMs, the next-byte distribution and the boundary-placement distribution can be disentangled and manipulated almost independently. That is a genuinely new framing, and the two proposed experiments are well-posed: distill language modeling only and check whether boundaries stay put; then flip it and change boundaries while self-distilling language modeling. If the hypothesis holds, it would make per-model tokenizers obsolete and turn byte-level transfer into an exact operation. The paper is honest about its status, the boundary divergence is defined clearly (1 - F1), and the author acknowledges the UTF-8 Latin bias and the risk that boundary changes reshape the backbone input.\n\nWhere it falls short is the evidence. The central hypothesis is motivated by 'initial measurements' that are not included: the Ukrainian embedding resettability check, the high-divergence settings between SentencePiece and BPE models, and the claim that Bolmo's predicted boundaries barely differ from the tokenizer it was byteified from. None of these appear in the paper, so a reader cannot evaluate them. The abstract says 'alongside preliminary measurements' but they are not there. That is a load-bearing gap, because the whole proposal rests on the plausibility of disentangling, and these are the only signs that it might work. Without the data, the claim is a stimulus, not a demonstration.\n\nI don't think the paper should be desk-rejected. The literature on tokenizer transfer and byte-level models is active, and this framing could reframe the way people think about boundaries. A serious referee could push the author to supply the missing measurements or at least present them in an appendix. The experiments are concrete enough to be run, and the author has the tools (Bolmo-style byteification) to do it.\n\nFor a reading group, I'd say yes if you want a sharp discussion of what counts as evidence in a position paper, and of whether the disentangling hypothesis deserves a test. I wouldn't cite it yet, because the central claims are unverified, but I'd cite it once the measurements are released.\n\nRecommendation: send it to peer review, with a clear request for the data behind the preliminary claims. If the author won't provide them, reject; if they do, this could become a useful roadmap paper.","headline":"A plausible and well-framed hypothesis, but the missing preliminary data make it a roadmap rather than a result; worth refereeing as a position paper.","tokens_in":4272,"tokens_out":2545,"would_cite":false,"duration_ms":25684,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Byte-model knowledge could transfer while boundary placement stays put","keywords":["byte-level language models","patch boundaries","boundary prediction","tokenizer transfer","distillation","next-byte prediction","disentanglement","model interface"],"falsifier":"Run the paper's first experiment: distill one byte-level model into another on next-byte predictions alone, choosing a teacher and student with strongly divergent boundary distributions. If the student's boundary divergence shifts substantially after distillation, or if the targeted capability fails to transfer, the disentangling hypothesis is falsified.","tokens_in":3594,"feed_emoji":"🧩","tokens_out":4081,"duration_ms":45107,"temperature":0.7,"pith_summary":"This paper argues that byte-level language models actually produce two separate distributions: one that predicts the next byte and one that decides where to place patch boundaries. The author hypothesizes these two can be disentangled and changed almost independently, so a model could absorb a teacher's skill without adopting its segmentation, or re-segment its input without losing knowledge. If true, the per-model tokenizer would no longer block capability transfer, making cross-tokenizer distillation exact and cheap. The paper proposes two concrete experiments to settle this, previewing the measurements needed to run them.","feed_headline":"Byte-model skills could transfer while boundaries stay put","feed_subtitle":"Two experiments would test whether next-byte knowledge and patch placement are independent in byte-level models.","key_machinery":"The joint distribution p(b, m) over a byte b and a boundary marker m, where every byte appears twice (plain and patch-ending), with marginals Σ_m p(b, m) and Σ_b p(b, m) as the language-modeling and boundary distributions. Bolmo emits this joint distribution, which makes the two marginals measurable and separately manipulable through distillation, carrying the entire disentangling argument.","core_discovery":"The paper's central claim is that the language-modeling distribution and the boundary-prediction distribution of a byte-level model are separable enough that each can be changed almost independently of the other. Using Bolmo as the canonical architecture, the author defines a joint distribution over a byte and a boundary marker, whose marginals are exactly the two distributions of interest. Because every byte-level model emits over the same 256-byte output space, capability transfer via next-byte distillation should be exact, not approximate. The two experiments are: distill only the language-modeling distribution to move a capability while checking the student's boundary divergence stays un","pith_inferences":["The same disentangling logic could extend to other jointly emitted structural distributions, such as word or sentence boundaries, not just patch boundaries.","Even a partial independence between the two distributions would meaningfully reduce the cost of adapting models to low-resource languages, where tokenizer mismatch currently forces expensive synthetic data pipelines.","The reported high divergence between SentencePiece- and BPE-derived boundary distributions suggests the proposed experiments already have ideal test pairs; a null result in one pair need not rule out disentangling in models with more separable boundary predictors.","If boundary retraining succeeds, compression rate becomes a practical efficiency lever for byte-level models, potentially lowering inference and memory costs without sacrificing capability."],"forward_implications":["Capability transfer between any two byte-level models becomes exact and routine, since both emit distributions over the same 256-byte alphabet.","Boundary placement becomes a direct control knob: retrain the boundary predictor to change compression rate without retraining the backbone transformer.","The per-model tokenizer is no longer an obstacle, eliminating the need for tokenizer transfer or extensive synthetic data generation.","Approximate cross-tokenizer distillation methods become unnecessary bridges, subsumed by the shared byte interface.","If the hypothesis holds, the community gains a strong structural reason to standardize on a byte-level interface for all language models."],"supporting_citations":[{"why":"Supplies Bolmo, the architecture whose joint output separates the two marginals, and the byteification method that makes the experiments possible.","marker":"Minixhofer et al., 2025a"},{"why":"Establishes entropy-based patch boundaries as the second distribution that byte-level models with patching produce.","marker":"Pagnoni et al., 2025"},{"why":"Shows boundary placement can be learned end-to-end, motivating the idea that boundaries are a manipulable output rather than a fixed tokenizer artifact.","marker":"Hwang et al., 2025"},{"why":"Demonstrates byte-level modeling without any aggregation, providing the baseline for a shared byte output space.","marker":"Xue et al., 2022"},{"why":"Offers the closest prior attempt—cross-tokenizer distillation through a temporary byte-level interface—which the paper supersedes by making the byte interface permanent.","marker":"Singh et al., 2026"},{"why":"Represents approximate cross-tokenizer distillation, which the paper's exact-transfer claim is positioned against.","marker":"Boizard et al., 2025"},{"why":"Provides the distillation mechanism that the first experiment uses to transfer capabilities over the language-modeling distribution.","marker":"Hinton et al., 2015"}],"fun_headline_variants":["Can byte-model skills transfer while boundaries stay fixed?","Disentangling byte-level learning from boundary decisions","Two experiments to test byte-level skill-boundary independence","Byte models: separating next-byte knowledge from patch placement"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claim rests on the assumption that a model's capabilities are stored in its deeper layers rather than its embeddings, so replacing embeddings with a byte interface does not erase them and the boundary predictor can be retrained without disturbing them.","fun_headline_variants_meta":{"raw":{"variants":["Can byte-model skills transfer while boundaries stay fixed?","Disentangling byte-level learning from boundary decisions","Two experiments to test byte-level skill-boundary independence","Byte models: separating next-byte knowledge from patch placement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1369,"prompt_tokens":675,"completion_tokens":694,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":419,"tokens_out":694,"duration_ms":7841,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:16:28.375572+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's first experiment: distill one byte-level model into another on next-byte predictions alone, choosing a teacher and student with strongly divergent boundary distributions. If the student's boundary divergence shifts substantially after distillation, or if the targeted capability fails to transfer, the disentangling hypothesis is falsified.","supporting_citations":[{"cited_title":"Cross-Tokenizer","cited_arxiv_id":null,"evidence_quote":"Offers the closest prior attempt—cross-tokenizer distillation through a temporary byte-level interface—which the paper supersedes by making the byte interface permanent."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents approximate cross-tokenizer distillation, which the paper's exact-transfer claim is positioned against."}],"review_version":1}