{"id":"6591da08-279b-409b-ab88-a0f78bab8d5a","arxiv_id":"2608.03839","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"A training-free speculative decoder using the verifier's hidden states as a semantic key retrieves correct continuations unreachable by exact suffix matching, lifting accepted length 24-29%.","lead":"Oilbird adds a second, semantic lookup to training-free speculative decoding: it re-keys the verifier's already-computed hidden states to retrieve correct continuations that exact text matching cannot reach. On tool-calling workloads it lifts accepted draft length by 24-29% inside existing drafters and reaches 4.4x autoregressive speed on API-Bank.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified: the addressing-vs-coverage claim is internally consistent and well-ablated; only the absence of public artifacts and a validation split warrants the conditional verdict.","rationale":"The reader's conditional verdict is appropriate, but the weakest assumption named — that cosine similarity at layer round(0.85L) generalizes across traffic — is not where I would place the main risk. The paper provides direct evidence for that assumption across ten benchmarks and two model families, and it explicitly limits the claim to repetitive tool-calling traffic. My stress-test found no internal inconsistency: the gap analysis, the retrieval design, the bonus-token lag in Eq. (3), the merge cost accounting, and the ablation magnitudes are mutually consistent. The strongest reason not to treat the headline numbers as settled is that the pipeline has several interacting hyperparameters and no public implementation or validation split, so the reported 24-29% transplant gains and 4.39x speedup could in principle reflect selection on the evaluation benchmarks. That is a reproducibility concern rather than a demonstrated flaw, so the verdict should remain CONDITIONAL rather than move to ACCEPT or REJECT.","tokens_in":20952,"tokens_out":20028,"duration_ms":235147,"concrete_test":"Run an independent re-implementation (or obtain the authors' code) and split API-Bank by request into a tuning set (first ~100 requests) and a held-out set; fit θ, k, the layer choice, Bret, and the Eq. (3) depth thresholds on the tuning set, then measure the +pool→+key accepted-length delta on the held-out remainder exactly as in Figure 4. If the held-out delta remains ≈24-29% and the API-Bank τ gain remains ≈+6%, the central claim is confirmed; if it falls materially, the headline gains are benchmark-tuned rather than mechanism-driven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"I could not identify a load-bearing flaw in the central argument. The diagnosis in Section 3 is measured directly on the traffic, with the gap defined by an exact-match oracle and the semantic key assessed at the same candidate budget. The deployed system's +6.2% accepted-length gain on API-Bank is quantitatively consistent with recovering ~80% of a 6.8% position-level gap, each backed by a ~6.4-token run, and the 24-29% transplant gains are controlled comparisons against each host's own exact-match key. The seemingly odd skip in Eq. (3) — copying from x_{i+2} rather than x_{i+1} — is explained by the bonus-token mechanism: the query state lags the committed bonus token, so the chain starts after the value while the verifier supplies the value itself. Ablations show no source is free to remove, and the semantic key is the smallest contributor, exactly as expected if it targets the ~7% gap. Limitations (memory, retrieval cost, batch-size sensitivity, task dependence) are disclosed. The only substantive caveat is evidential: no code/data are released, and hyperparameters (θ, layer, Bret, depth thresholds) are not tied to a validation split, so the reported numbers are not yet independently checkable. That warrants CONDITIONAL, not rejection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Oilbird, a training-free speculative decoding extension that augments an exact-suffix lexical drafter with a second, per-position semantic key over hidden states the verifier already computes. It first presents a method-independent diagnosis on ten benchmarks, separating coverage failures (continuation absent from the pool) from addressability failures (present but unreachable by exact matching). On API-Bank the paper measures a 6.8% identifiability gap and reports that a cosine key over layer round(0.85L) hidden states recovers about 81% of those gap positions at the same candidate budget. The deployed system merges the semantic source into a multi-source tree within a fixed node budget, reports +6.2% accepted length over its lexical-only host on API-Bank, +24–29% accepted length when transplanted into three published drafters, and 4.39x autoregressive speedup on API-Bank with Llama-3.1-8B, ahead of the training-free baselines and EAGLE-3 in the same harness. The paper includes ablations, batch-size probes, memory/retrieval-cost accounting, and an ethics statement.","tokens_in":21320,"tokens_out":12481,"duration_ms":145418,"significance":"The central contribution is an empirical diagnosis that is falsifiable and largely method-independent: the no-drafter census, the exact-match oracle, bootstrap intervals, and token-for-token audited greedy verification are credible controls. If the results hold, they reframe training-free speculative decoding on repetitive serving traffic from a coverage problem to an addressing problem, and show that a cheap side channel already computed by the verifier can close a substantial part of that gap. The transplant study into three drafters the authors did not write is a strong control, and the ablations consistently show that no source is free to remove. The paper is also appropriately careful about scope: it states where the semantic source is neutral or negative, discloses memory and retrieval costs, and reports batch-size sensitivity. The main weaknesses are evidential rather than technical: key hyperparameters are fixed without a validation split, and no artifacts are released. I agree with the stress-test note that no load-bearing flaw in the central derivation or measurement was identified; the off-by-one in Eq. (3) is plausibly explained by the bonus-token lag, and the semantic","major_comments":[{"comment":"Several load-bearing parameters are fixed without a validation split: theta = 0.8, layer round(0.85L), k = 8, Bret = 16/60, and the depth schedule D(s) in Eq. (3). The central quantitative claims (81% gap recall, +6.2% over the lexical-only host, +24-29% transplant gains, and the Table 1 speedups) are measured on the same benchmarks used to motivate these settings. This leaves open the possibility that part of the reported gain is a selection artifact. I request either a held-out validation split or a sensitivity sweep over theta, layer, k, and Bret showing the results are stable within a reasonable operating range.","section":"Algorithm 1, Eq. (3), §5"},{"comment":"The diagnosis in §3 is defined over 'everything the requests before it contain' and Table 7 says 'verbatim in an earlier trace,' while §4 states that the deployed store records only generated tokens and that prompts are never stored. If the census predicate includes prompt and tool-observation text that the deployed pool excludes, then the 6.8% identifiability gap and the 81% semantic recall would overestimate what Oilbird can recover. Please state unambiguously whether the census store is identical to the deployed store, and if it is not, recompute the gap and recall under the deployed store definition. This is load-bearing for the addressing-vs-coverage claim.","section":"§3 vs. §4"},{"comment":"The paper states that an audit script recomputes 765 numbers from raw data, but no code, data, or raw logs are released. For an empirical systems paper whose central claims are measured numbers (4.39x, +24-29%, 6.8% gap), this blocks independent verification of the very results on which the contribution rests. Please release the implementation, the exact-match oracle, the audit script, and per-benchmark logs, or provide a detailed reproducibility appendix with the same information.","section":"§5, Table 2, audit statement"}],"minor_comments":[{"comment":"SELFCAL is referenced in Algorithm 1 and in the component ablation, but its muting criterion is never defined. Please specify the mechanism or, if it is a minor safeguard, state that it is disabled in the reported configuration.","section":"Algorithm 1, line for SELFCAL"},{"comment":"The sentence 'With both lexical sources removed it stops drafting altogether, committing exactly one token per verification pass' appears inconsistent with Table 5(a), which reports tau = 3.75 for the 'semantic only' row. Please clarify the configuration being described in the text.","section":"§6.1 vs. Table 5(a)"},{"comment":"The ablation bars in Figure 3 are reported without repeat variance. Since Table 2 mentions two repeats and the authors already use bootstrap intervals elsewhere, adding error bars or per-replicate values would strengthen the claim that the semantic key's loss is 'never zero and never favorable.'","section":"Figure 3"},{"comment":"Wilson intervals are only given for the Llama API-Bank row, and the authors note that request-level resampling widens them. Reporting request-resampled intervals for the other high-gap rows (e.g., tau-bench retail, ToolAlpaca) would help readers gauge how robust the cross-benchmark recall numbers are.","section":"Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the empirical design is strong. The main risk is not internal inconsistency but the lack of a validation split and public artifacts; if the authors supply those, I expect the results to hold. The store-definition ambiguity in §3/§4 should also be resolved before acceptance, since it bears directly on the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one. The paper's core move is separating two failure modes of training-free speculative decoding on tool-calling traffic: the next token is either absent from the pool or present but unreachable by exact suffix matching. They measure that on real traffic and find about half of the misses are the second kind, an addressing gap, not a coverage gap. That diagnosis is the actual contribution, and it is backed by a clean no-drafter census: an exact-match oracle, a static-embedding control, and a per-position hidden-state key that recovers 81% of the gap positions on API-Bank. I believe these numbers. The deployment story is also solid: the semantic key is merged into an existing tree at a fixed node budget, verified token-for-token against greedy decoding, and transplanted into three published drafters with matched pools and budgets, lifting accepted length by 24-29%. The ablations are honest - no source is free to remove, the semantic key is the smallest contributor, and the merge itself matters more than the key. They even show the method degrades at batch size 32. That is a well-executed, well-scoped paper.\n\nThe soft spots are evidential, not logical. No code or data are released, so all speedup and accepted-length numbers rest on the authors' harness. Worse, the hyperparameters in Algorithm 1 - theta, the 0.85L layer, depth thresholds, the 16-of-60 node budget - are not tied to a validation split. The paper uses one frozen configuration for all results, but there is no evidence those choices were not tuned on the test workloads. The bootstrap intervals in Table 8 are flagged as optimistic because positions are not independent, and the authors acknowledge this. Also the method is task-dependent by design: it wins on repetitive tool-calling traffic, is neutral elsewhere, and loses to EAGLE-3 on non-repetitive workloads. Memory overhead (~8 KB per token) is disclosed but not deeply analyzed.\n\nI do not think there is a load-bearing flaw in the central argument. The addressing-vs-coverage claim is internally consistent and the quantitative consistency between the 6.8% gap, 81% recovery, and the deployed +6.2% accepted-length gain checks out. The absence of artifacts and a validation split is what makes this conditional rather than definitive: the reported 4.4x speedup should be re-measured by an independent implementation.\n\nThis deserves a serious referee. The right outcome is major revision with release of code, data, and a validation-split analysis for the hyperparameters. If that is fixed, this becomes a reference point for training-free speculative decoding on structured traffic. I would bring it to a reading group on efficient inference and would cite it if I worked in that area.","headline":"A well-measured, genuinely novel fix for a specific failure of training-free speculative decoding on tool-calling traffic; the central claim holds in scope, but the missing artifacts and unvalidated hyperparameters make it a conditional accept.","tokens_in":21792,"tokens_out":972,"would_cite":true,"duration_ms":13779,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training-free speculative decoding fails on tool-calling traffic because its exact-match key cannot address continuations already in its own store; Oilbird re-keys the same store by hidden states the verifier already computes and recovers m","keywords":["speculative decoding","training-free drafting","tool calling","hidden-state retrieval","semantic key","identifiability gap","lossless acceleration","draft tree merging"],"falsifier":"On a fresh tool-calling benchmark with high lexical availability, run the paper's exact-match oracle and then the semantic key at the same eight-candidate budget: if recall at gap positions does not clearly beat the static token-embedding baseline (~23%) and approach ~81%, the addressing diagnosis fails. On API-Bank itself, zeroing the semantic entry budget should cost about +6.2% of accepted length; a null result there would falsify the practical claim.","tokens_in":20905,"feed_emoji":"🔑","tokens_out":7344,"duration_ms":65760,"temperature":0.7,"pith_summary":"Copy-based speculative drafters propose tokens by matching an exact suffix of the current context against a store of previously generated text. On tool-calling traffic, the right continuation is often already in the store, but a single newly minted value—a name or ID—breaks every suffix ending at that token, so the exact-match key cannot reach it. The paper measures that about half of what the strongest exact-match drafter misses on API-Bank is present yet unreachable, and calls this the identifiability gap. Oilbird closes part of it with a second key over the same store: the hidden state the verifier has already computed at each committed token, retrieved per position by cosine similarity and merged into the existing lexical draft tree. Under matched pool and node budget it recovers the true next token at four in five gap positions, raises accepted length by 24–29% inside three published drafters, and reaches 4.39x autoregressive speed on API-Bank with Llama-3.1-8B.","feed_headline":"Hidden-state keys lift copy-based decoding by up to 29%","feed_subtitle":"Re-keying the draft store by model states reaches continuations exact matching leaves in the pool.","key_machinery":"One store, two keys: over a pool of previously generated tokens the system keeps a longest-suffix index and a cosine index over hidden states read at layer round(0.85L). The semantic key's query is the hidden state h_t left by the last verifying forward; it retrieves the top-8 pool positions by cosine similarity above a firing floor θ=0.8 and copies a chain γ_i = x_{i+2:i+1+D(s_i)} from two positions past each neighbor, where depth D(s) is 48, 8, or 0 tokens according to similarity. The merge re-walks the draft tree from the root, costing |γ|−π(γ,T) nodes where π is the longest prefix already present, so semantic chains ride inside the lexical tree at a fixed 16-node reservation of the 60-no","core_discovery":"The paper's central claim is that the bottleneck in training-free speculative decoding on repetitive tool-calling traffic is addressing, not coverage. A position-by-position census over ten benchmarks shows that on API-Bank about half of what the strongest exact-match drafter misses is text already verbatim in the pool yet unreachable by suffix matching: the failure is a cliff, not a slope, with median matched suffix length dropping from sixteen tokens at successful positions to one at gap positions. The proposed fix, Oilbird, stores one hidden vector per generated token—captured for free from the verifying forward—and queries it per position by cosine similarity. That key finds the true nex","pith_inferences":["The identifiability gap should generalize to any traffic where a few minted values sit inside repeated structured frames—agent logs, SQL, configs, JSON envelopes—so the same re-keying diagnosis may transfer to other cache-based reuse systems, though the paper only measures language decoding.","Because the key matches hidden states rather than strings, it may also reach positions where the text differs but the situation is semantically aligned; the per-position similarity-scaled depth hints at this, but the paper does not test paraphrase-level reuse separately.","The measured near-independence of the semantic key and the lexical keys (lift 1.013 and 1.009) suggests adding a third orthogonal addressing key could keep buying accepted length until the uniquely-addressable positions are saturated; that saturation point is not identified.","Layer choice is an assumption with an easy check: the 0.85L layer works for the two 8B families measured, and recall at gap positions on a new family or scale should be probed before trusting the 81% figure."],"forward_implications":["A larger store is not the remedy the numbers point to: on API-Bank, about half of the strongest exact-match drafter's misses are already in the pool, so re-keying that pool can buy more than growing it.","The semantic source is portable: three published training-free drafters, each already holding the same pool under its own exact-match key, gain 23.6–29.4% accepted length when the semantic key is merged in at an unchanged node budget.","Oilbird is lossless and training-free, and on API-Bank with Llama-3.1-8B it beats the trained EAGLE-3 in wall-clock speedup (4.39x vs 2.03x) while producing identical greedy output.","The gain is regime-specific: at batch sizes 8–32 speculation's edge over autoregressive decoding shrinks and the semantic key's per-position retrieval is neutral or negative, so the result is a batch-1 serving speedup.","On non-repetitive workloads the semantic source is neutral rather than costly, and a trained drafter remains the better choice there; the method's value concentrates on repetitive structured traffic."],"supporting_citations":[{"why":"Defines speculative decoding and the lossless verification property Oilbird inherits.","marker":"Leviathan et al., 2023"},{"why":"Gives the sampling-based acceptance rule and losslessness under distribution preservation.","marker":"Chen et al., 2023"},{"why":"Introduces tree-based verification with attention masks that Oilbird's merged tree relies on.","marker":"Miao et al., 2024"},{"why":"Extends tree speculation (Sequoia) used for scoring multiple draft branches in one forward pass.","marker":"Chen et al., 2024"},{"why":"SuffixDecoding is the strongest exact-match baseline and supplies the online pool the semantic key re-indexes.","marker":"Oliaro et al., 2025"},{"why":"Token Recycling supplies the token-adjacency draft source and the idea of harvesting verification by-products.","marker":"Luo et al., 2025"},{"why":"kNN-LM establishes nearest-neighbor retrieval over hidden states, the origin of the semantic key.","marker":"Khandelwal et al., 2020"},{"why":"GOOSE is the multi-source tree decoder Oilbird merges into and the direct baseline.","marker":"Jin et al., 2026"},{"why":"EAGLE-3 is the trained drafter reference Oilbird must beat on speed.","marker":"Li et al., 2025"},{"why":"ToolSpec is a transplant host where the semantic key is shown to add 27.4% accepted length.","marker":"Xia et al., 2026b"}],"fun_headline_variants":["Hidden-state keys beat exact-match for speculative decoding","Re-keying draft pool by model states lifts speed 24-29%","Suffix matching fails on tool calls; hidden-state keys fix it","Addressing, not coverage: hidden-state keys win in speculative decoding","Oilbird: verifier keys reclaim drafts exact matching leaves behind"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claim rests on cosine similarity between hidden states taken at layer round(0.85L) of the target model identifying past positions whose next token is the correct continuation of the current context; if that similarity stops tracking correct continuations on other traffic, the recovered gap and the 24–29% gains disappear.","fun_headline_variants_meta":{"raw":{"variants":["Hidden-state keys beat exact-match for speculative decoding","Re-keying draft pool by model states lifts speed 24-29%","Suffix matching fails on tool calls; hidden-state keys fix it","Addressing, not coverage: hidden-state keys win in speculative decoding","Oilbird: verifier keys reclaim drafts exact matching leaves behind"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0011,"raw_usage":{"total_tokens":4420,"prompt_tokens":730,"completion_tokens":3690,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":3602}},"tokens_in":474,"tokens_out":3690,"duration_ms":26564,"temperature":1.0,"reasoning_tokens":3602,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:14:09.378074+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fresh tool-calling benchmark with high lexical availability, run the paper's exact-match oracle and then the semantic key at the same eight-candidate budget: if recall at gap positions does not clearly beat the static token-embedding baseline (~23%) and approach ~81%, the addressing diagnosis fails. On API-Bank itself, zeroing the semantic entry budget should cost about +6.2% of accepted length; a null result there would falsify the practical claim.","supporting_citations":[],"review_version":1}