{"id":"d8460713-59b0-4ca1-b75b-ba29c1e30e10","arxiv_id":"2505.20438","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HAMburger generates multiple output tokens per LLM forward pass via a compositional embedder and a micro-step decoder, achieving up to 2x throughput with one KV slot per group.","lead":"HAMburger lets a language model generate several tokens in one forward pass by smashing them into a single cache entry. This can cut KV memory and boost tokens-per-second by up to about 2x on a 1B model, potentially lowering serving costs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central quality claim depends on an untested information-capacity assumption: fused low-entropy token groups must be reconstructable exactly by the micro-step decoder, with no verification to catch errors.","rationale":"The reader's weakest_assumption identifies the same central risk: the method assumes low-conditional-entropy tokens can be generated accurately from local information, so a single fused KV cache suffices. My review agrees that this is the load-bearing condition. The alternative concern, that the 'sub-linear growth' wording is unsupported because the micro-step size is bounded by MAX_STEPS=4, is real but does not threaten the method's practical viability; it is an asymptotic overstatement. The information-capacity assumption, by contrast, is what the entire quality guarantee rests on, and it is the least supported part of the paper. The evidence offered is an ablation table showing per-token micro-step accuracy of about 93% on three tasks, but no exact-segment accuracy, no error injection study, and no variance or confidence intervals. For a method that blindly trusts its own drafts with no verification, even a small per-token error rate can compound over the response. The proposed reconstruction test directly measures whether the fused representation preserves enough information to reproduce the segmented tokens exactly. If it does, the method's central claim is credible. If it does not, the reported downstream scores at tuned confidence thresholds may be concealing task- or prompt-specific failures. I therefore keep the CONDITIONAL verdict: the paper should be accepted only if this experiment is added and the results support the information-capacity assumption, or if the claims are weakened to match the evidence.","tokens_in":16929,"tokens_out":5430,"duration_ms":59218,"concrete_test":"Run a held-out reconstruction test on the same evaluation domains (e.g., GSM8K and HumanEval). Using the trained HAMburger model and the entropy-based segmentation, for each segment record the micro-step decoder's greedy prediction for the non-first tokens given only the fused hidden state, and compare against the base model's full-context greedy continuation. Report (a) per-token accuracy and, crucially, (b) exact-match rate over entire segments. If exact-match rate is high (>=98%) across segments, the information-capacity concern is largely mitigated; if it is closer to the token-accuracy level implied by Table 2, then a non-negligible fraction of macro-steps contain errors, and the paper must additionally measure downstream score with single-token decoding forced, to quantify error propagation. This single experiment would settle whether the 'maintaining quality' claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"HAMburger's central claim is that it can generate multiple tokens per macro-step 'while maintaining quality' because low-conditional-entropy tokens can be predicted from a single fused KV cache. Two load-bearing conditions must hold. (1) The compositional embedder's fused hidden state preserves enough information for the micro-step decoder to reproduce the non-first tokens of each segment exactly. (2) Since there is no rejection or verification of self-drafted tokens, any micro-step error becomes part of the KV cache and context, so errors propagate into subsequent macro-steps. Neither condition is established. Table 2 gives 'Eval > 1 Token Acc' of 0.93 for the final model; for a 3-token micro-step, this implies roughly 20% of macro-steps contain at least one erroneous non-first token if errors are independent. The paper reports aggregate task scores at a tuned confidence threshold but no per-segment exact-match rate, no measure of how often an error in a fused segment changes the final answer, and no variance. The Section 5 admission that 'the compositional embedder is not perfect' acknowledges the limitation but does not quantify its downstream cost. Low conditional entropy under the base model's full-context predictions does not imply low reconstruction error from a compressed average hidden state; code and math tokens require exact spelling, so the mapping from fused representation to exact token sequence is the fragile link. Thus the central quality-maintenance claim currently rests on an unverified assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HAMburger, an inference-time augmentation of a base LLM with a compositional embedder and a micro-step decoder. At each macro-step, the embedder fuses the tokens generated in the previous step into a single hidden state, the base model performs one forward pass and stores one KV entry for the fused group, and the micro-step decoder autoregressively emits several tokens plus a stop decision. The method is trained by SFT on data segmented according to the base model's own conditional entropy. The authors claim that this shifts KV-cache and forward-FLOP growth from linear to sub-linear in output length, reduces KV cache computation by up to 2x, improves decoding TPS by up to 2.2x, and maintains or improves quality on short- and long-context tasks.","tokens_in":17179,"tokens_out":6170,"duration_ms":61892,"significance":"If the empirical claims held, HAMburger would be a useful contribution: it is a self-speculative scheme with no verification pass, constant drafting overhead, and a KV-compression mechanism that is orthogonal to prior cache-compression and offloading work. The component ablations in Table 2 are informative, and the use of a public data mix is a strength. However, the headline asymptotic claim is unsupported, and the quality-maintenance claim rests on a small number of aggregate point estimates from a single 1B model. After correction, the contribution is best characterized as a constant-factor acceleration and compression method rather than a change in asymptotic complexity.","major_comments":[{"comment":"The claim that KV cache and forward FLOPs grow sub-linearly with output length is not supported. In Eq. (1), n is the number of micro-step tokens per macro-step, and training fixes a maximum micro-step size of 4 (Section 4.1). With n bounded by a constant, the number of KV entries is L/n_avg = Θ(L), and total attention FLOPs remain Θ(L^2) with a smaller constant; no sub-linear scaling follows. The paper should either show that the stop head produces an unbounded and growing n in practice, or replace the asymptotic claim with a constant-factor improvement.","section":"Abstract; §3.7, Eq. (1)"},{"comment":"The central 'maintaining quality' claim is not established. Table 2 reports a per-token accuracy of 0.93 for non-first micro-tokens, but the paper never reports the per-segment exact-match rate or the downstream effect of a wrong non-first token on the final answer. For a 4-token micro-step, independent per-token errors at 7% would put at least one error in roughly 20% of macro-steps; since HAMburger blindly trusts drafted tokens (Section 3.7) and Section 5 admits that the compositional embedder is not perfect, the paper needs to quantify how often fused segments are faithfully reconstructed and how reconstruction errors propagate into subsequent macro-steps.","section":"Table 2; §4.2; §5"},{"comment":"The evaluation is too thin to support the cross-task quality claim. All results come from a single Llama-3.2-1B model with greedy decoding and point estimates, with no error bars or multiple seeds. Comparisons to BLT-8B and to StreamingLLM/TOVA/Quest use different base models and, in the latter case, a single sub-task; there is no comparison to Medusa or EAGLE on the same base. Without these, the statement that HAMburger 'maintains and even surpasses' the base model is a point-estimate observation rather than a demonstrated property.","section":"§4.1, §4.2, Table 2"},{"comment":"The confidence threshold is a free parameter that directly sets the measured compression and speedup, and the paper does not specify a single operating point for the headline claims. Figures 3 and 4 show quality varying with the threshold, so 'maintaining quality while achieving up to 2x' requires reporting quality and speed jointly at one preset threshold; otherwise the two numbers may be selected from different thresholds.","section":"§3.5, §4.2, Figures 3–5"}],"minor_comments":[{"comment":"There are several typos and wording issues: 'Hierachically' should be 'Hierarchically', 'switches between two different phrases' should be 'phases', and 'we don not need' should be 'we do not need'.","section":"§1, §3.1, §3.5"},{"comment":"The dynamic segmentation rule is described only heuristically ('global statistics', 'predefined threshold', 'low entropies relative to that of the first token'). Please provide the exact algorithm or pseudo-code so the data preparation is reproducible.","section":"§3.5"},{"comment":"The metric 'Eval > 1 Token Acc' should be defined precisely, including which tokens are in the denominator and how the average is computed across segments, so the reader can relate it to per-segment exact-match accuracy.","section":"Table 2"},{"comment":"The speculative-decoding baselines are not fully specified; the paper should state the draft model, gamma values, and hardware/software settings for the TPS numbers, since efficiency claims are system-dependent.","section":"§4.3, Figure 5"},{"comment":"The comparison to StreamingLLM, TOVA, and Quest uses a 7B model with a 4096-token KV budget against the authors' 1B HAMburger model; this is an apples-to-oranges comparison and should be labeled as such, or matched in base model and budget.","section":"§4.2.2, Figure 4"},{"comment":"The bottom x-axis label 'KV Cache Compression' is presented as a percentage; please clarify whether a value like 187% means 1.87x compression, 187% of baseline KV size, or another convention.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope. The sub-linear-growth claim is clearly overreaching and should be corrected, and the quality-maintenance claim needs the additional per-segment and variance analyses described in the major comments. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my quick take on HAMburger. The actual new thing is the specific combination: an entropy-based dynamic segmentation that fuses multiple tokens into one KV via a cross-attention embedder, a small micro-step decoder that generates several tokens per macro-step, and a stop head that decides when to stop—all without a verification pass. That combination is not in BLT, MegaByte, or Medusa, and it's a clever way to get constant-factor speedups on small models, where draft models are hard to align. The ablation in Table 2 is genuinely useful—it shows each piece (middle hidden features, cross-attention embedder, stop head) contributes.\n\nWhat I don't buy is the \"sub-linear growth\" claim in the abstract and Section 3.7. With MAX_STEPS=4, KV cache entries grow linearly with output length, just with a smaller constant. The formula r = n·C(S)/(C(S)+(n−1)c) treats n as a variable, but it's bounded. So the asymptotic framing is wrong, and the paper should say \"reduced constant\" not \"sub-linear.\"\n\nThe evaluation is the bigger concern. It's one 1B model, no error bars, and no Medusa/EAGLE comparison on the same base—those are the obvious baselines. BLT-8B is a strange comparison for a 1B model. The reported 0.93 \"Eval > 1 Token Acc\" is decent, but for 3-token micro-steps it implies a meaningful fraction of macro-steps contain at least one wrong non-first token, and since you blindly trust them, those errors enter the context. The paper admits the compositional embedder isn't perfect but doesn't quantify the downstream cost. This isn't a fatal flaw—it's an empirical question that a revised evaluation could answer—but it's the load-bearing assumption.\n\nNet: this is a solid incremental contribution to inference acceleration, aimed at people working on serving small models and long-context. It deserves peer review. I'd ask the authors to fix the asymptotic claim, add variance and strong baselines, and either show error analysis or add a rejection/verification option.","headline":"A plausible self-speculative decoder that fuses low-entropy tokens for ~2x speedup on a 1B model; the sub-linear growth claim is overstated and the evaluation needs stronger baselines.","tokens_in":17747,"tokens_out":2722,"would_cite":true,"duration_ms":26456,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token smashing fuses multiple tokens into one KV cache, giving up to 2× faster LLM decoding while keeping quality.","keywords":["LLM inference","KV cache compression","multi-token decoding","speculative decoding","conditional entropy","token fusion","hierarchical autoregressive model"],"falsifier":"Run HAMburger on a generation task where every token is hard to predict from local context — e.g., emitting random strings, exact long identifiers, UUIDs, or adversarially chosen sequences — and measure both quality and the average macro-step length. If the micro-step decoder is forced back to one token per step whenever entropy is high, the claimed sub-linear growth of KV cache and FLOPs will not survive on such inputs.","tokens_in":16645,"feed_emoji":"🍔","tokens_out":5773,"duration_ms":47708,"temperature":0.7,"pith_summary":"The paper argues that LLM decoding wastes compute and memory by treating every token as equally costly, giving each token its own forward pass and its own KV cache entry. HAMburger instead has the model itself decide which tokens are easy and can be generated confidently from local information, groups those tokens into a single macro-step, and stores only one fused KV cache for the group. The claim is that such token smashing cuts the growth of KV cache and forward FLOPs from linear to sub-linear in output length, delivering up to a 2× reduction in KV cache computation and up to 2× tokens per second while matching or improving quality. If true, this makes long-context and high-throughput serving cheaper without a separate draft model or lossy KV compression.","feed_headline":"Token smashing gives up to 2× faster LLM decoding","feed_subtitle":"HAMburger fuses low-entropy tokens into a single KV cache, cutting compute and memory while keeping quality.","key_machinery":"The two grafted modules carry the argument. The relative-position-aware compositional embedder, built as a Perceiver-style cross-attention whose query is the mean of the input token embeddings, maps a list of token embeddings into a single hidden state that is permutation-sensitive; it also offsets position information so the base model knows how many tokens were fused. The micro-step decoder is a small stack of transformer layers that conditions on hidden states from middle layers and the last layer to autoregressively emit micro-step tokens, and a binary stop head decides whether another micro-step or a new macro-step should happen. Around them, dynamic data segmentation uses the base model's conditional entropy to split training responses into segments whose non-first tokens have low entropy, so the model learns exactly what dose of information a single KV cache can hold.","core_discovery":"The central discovery is that a base LLM can be taught to self-segment its own output: a compositional embedder fuses an ordered list of tokens from the previous macro-step into a single embedding, and a micro-step decoder rolls out a small run of tokens plus a binary stop decision before another forward pass of the base model is needed. The segmentation is guided by the base model's own conditional entropy, so tokens that the model is confident about are generated micro-step by micro-step from one set of KV caches, while the first token of each segment is handled by the full base-model forward. HAMburger therefore functions as a self-speculative decoder that blindly trusts its own drafts and pays no verification forward pass. The paper reports that this maintains task quality on instruction following, math, reasoning, code, and long-context benchmarks while compressing KV cache computation by up to 2×.","pith_inferences":["The 'easy token' hypothesis suggests a deeper architectural change: future LLMs could be trained from scratch with variable-rate decoding, where each KV cache entry is annotated with the amount of information it stores, rather than patched on after the fact.","A testable extension would be to use HAMburger's own predicted stop-head confidence as a calibration signal for per-token uncertainty, turning the fused-cache mechanism into a cheap probe for model knowledge or hallucination risk.","The reported speedups are on a 1B model; the ratio of base-model cost to micro-step cost suggests larger bases could see larger absolute savings, but also larger risk if the micro-step decoder's errors compound."],"forward_implications":["KV cache computation and storage drop by up to 2× on short- and long-context generation, with a tunable confidence threshold trading a little quality for more compression.","KV cache and forward FLOPs grow sub-linearly with output length instead of linearly, which matters most when generations or contexts are long.","The method works with small base models, a regime where speculative decoding struggles because a fast enough draft model is hard to find.","Because it is batch-friendly and hardware-agnostic, the speedup can be combined with orthogonal KV compression or offloading techniques.","At inference time the speed adapts to the prompt: easy, low-perplexity queries get more tokens per step and hard queries fall back to single-token steps."],"supporting_citations":[{"why":"Supplies the patch-level decoding approach that HAMburger contrasts with: BLT fixes byte patches and needs a separate entropy model, while HAMburger self-segments token patches.","marker":"[31]"},{"why":"Defines the speculative decoding paradigm whose drafting, verification, and acceptance-rate limits HAMburger claims to bypass with blind self-drafting.","marker":"[5]"},{"why":"Evidence that tokens do not all need global context, the core premise behind fusing low-entropy tokens into one KV cache.","marker":"[21]"},{"why":"Inspires the position-offset design that keeps the base model aware of fused-token count, and supports the idea of token importance estimation.","marker":"[25]"},{"why":"The cross-attention mechanism the compositional embedder is based on for extracting a single representation from a set of token embeddings.","marker":"[18]"},{"why":"The base model family on which HAMburger is built and fine-tuned, setting the vocabulary, tokenizer, and architecture constraints.","marker":"[14]"}],"fun_headline_variants":["HAMburger smashes tokens, doubles LLM decoding speed","Smash tokens, not quality: HAMburger 2× TPS","Token smashing halves KV cache, 2× faster LLM","One KV cache, many tokens: HAMburger 2× TPS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole speedup rests on the premise that tokens the base model deems low-entropy can be predicted correctly from local context alone, so that one fused KV cache holds enough information for the micro-step decoder to produce several correct tokens.","fun_headline_variants_meta":{"raw":{"variants":["HAMburger smashes tokens, doubles LLM decoding speed","Smash tokens, not quality: HAMburger 2× TPS","Token smashing halves KV cache, 2× faster LLM","One KV cache, many tokens: HAMburger 2× TPS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001023,"raw_usage":{"total_tokens":4340,"prompt_tokens":995,"completion_tokens":3345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":3280}},"tokens_in":611,"tokens_out":3345,"duration_ms":25330,"temperature":1.0,"reasoning_tokens":3280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:54:30.384341+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run HAMburger on a generation task where every token is hard to predict from local context — e.g., emitting random strings, exact long identifiers, UUIDs, or adversarially chosen sequences — and measure both quality and the average macro-step length. If the micro-step decoder is forced back to one token per step whenever entropy is high, the claimed sub-linear growth of KV cache and FLOPs will not survive on such inputs.","supporting_citations":[{"cited_title":"Pagnoni, R","cited_arxiv_id":null,"evidence_quote":"Supplies the patch-level decoding approach that HAMburger contrasts with: BLT fixes byte patches and needs a separate entropy model, while HAMburger self-segments token patches."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the speculative decoding paradigm whose drafting, verification, and acceptance-rate limits HAMburger claims to bypass with blind self-drafting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Evidence that tokens do not all need global context, the core premise behind fusing low-entropy tokens into one KV cache."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Inspires the position-offset design that keeps the base model aware of fused-token count, and supports the idea of token importance estimation."},{"cited_title":"Grattafiori, A","cited_arxiv_id":null,"evidence_quote":"The base model family on which HAMburger is built and fine-tuned, setting the vocabulary, tokenizer, and architecture constraints."}],"review_version":1}