L$^3$: Large Lookup Layers

Albert Tseng; Christopher De Sa

REVIEW 3 major objections 6 minor 2 references

L$^3$: Large Lookup Layers

T0 review · 3 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large Lookup Layers (L3), which cache per-token embeddings inside transformer layers, beat dense models and Mixture-of-Experts models at equal active parameters and FLOPs.

desk verdict A promising new static-routing sparse layer with solid dense-vs-L3 evidence, but the headline claim against MoEs rests on baselines that look undertuned. read the letter →

arxiv 2601.21461 v3 pith:FKRBCYH4 submitted 2026-01-29 cs.LG cs.AI

Albert Tseng , Christopher De Sa This is my paper · ORCID

classification cs.LGcs.AI

keywords L3layersparselanguagemodelsMixture-of-ExpertsembeddingallocationLZWcompressionstaticroutingmodelsparsityCPUoffloading

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces the Large Lookup Layer, a sparse transformer layer that acts like a giant token-specific embedding table placed inside the model. Instead of dynamically routing tokens to experts as MoEs do, L3 uses the token ID to select a fixed set of learned embeddings; the token's hidden state then attends to those embeddings, making the lookup context-dependent while the routing stays static. The authors show that adding one or two such layers to transformers with up to 2.6B active parameters improves language modeling perplexity and downstream accuracy relative to dense models and to equally sparse, equally costly MoEs, with gains visible from early in training and no auxiliary routing losses. A compression-based allocation algorithm (a variant of LZW) decides how many embeddings each token gets, and the paper reports it roughly doubles the perplexity improvement over a uniform allocation. A sympathetic reader would care because L3 opens a new axis of sparsity that is systems-friendly: the parameters for the next token are known in advance, enabling CPU offloading and prefetching with minimal overhead.

What carries the argument

The central object is the Large Lookup Layer (L3), a transformer layer parameterized by per-token-ID key matrices K_t and value matrices V_t, an up-projection, and a mixing matrix. The forward pass computes softmax(K_t x) to attend over the token's allowed embeddings, aggregates the corresponding values, projects them up, and mixes the result with the residual stream. The embedding allocation—how many embeddings each token gets, chosen by a variant of the LZW compression algorithm that favors frequent suffixes and caps at k per token—is the main quality-versus-speed knob. The static routing property is what gives L3 its systems advantage: active parameters are known at token generation time,

What would settle it

Train the same-size MoE baselines with a thorough hyperparameter sweep and a standard token budget (e.g., 5–10x more tokens than used here) and compare final perplexity on a held-out language modeling corpus; if an MoE then matches or beats the L3 model at equal FLOPs and sparsity, the paper's central claim fails. Alternatively, retrain the L3 model with a uniform per-token embedding allocation and check whether it still beats the dense baseline: if it does not, the LZW allocation is the true source of the gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that static, token-based routing can deliver the modeling benefits of dynamic expert routing without its systems costs. An L3 layer holds key and value embeddings for each token ID; given a token, the hidden state scores that token's allowed embeddings via attention, aggregates the corresponding values, projects them up, and mixes the result into the residual stream. Because the set of active embeddings is known as soon as the token ID is generated, the layer's parameters can be fetched from CPU during earlier computation, and training can be organized as block-diagonal attention over sorted token batches. The authors train transformers at 800M, 1.5B, and 2.6B ac

Load-bearing premise

The load-bearing premise is that the Mixture-of-Experts baselines are configured and tuned representatively, since the central claim that L3 strongly outperforms iso-sparse MoEs relies on those baselines; a secondary premise is that the LZW allocation, fit to a 1GB slice of the training corpus, transfers to the evaluation distribution.

Editorial extensions

If this is right

Sparse scaling can come from static lookup layers instead of dynamic routers, eliminating auxiliary routing losses and load-balancing instability.
L3 layers effectively cache frequent token-sequence information, letting the model skip decoder computation; tuned-lens KL drops at L3 layers support this interpretation.
Inference can keep most parameters on CPU: the paper reports that with L3 placed after four decoder layers, offload latency is fully masked at batch size 1.
Since L3 targets different per-layer sparsity rates and modeling behavior than MoEs, the paper proposes them as a complementary axis that could be combined with MoEs.
Weight tying between the key and value embeddings preserves quality while roughly halving data transfer and improving effective sparsity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the static-routing result holds at larger scales, L3 could be a cheap way to add capacity to existing dense or MoE models without increasing inference FLOPs, since total parameters can be large while active parameters stay tiny.
The LZW allocation suggests deeper ties between lossless compression and neural architecture; one could learn the allocation end-to-end or adapt it per domain, potentially improving the gap further.
Because routing depends only on token ID, L3 is a natural fit for low-memory or edge deployment: a model with tens of billions of parameters could run on a single GPU by keeping lookup tables on CPU.
The paper's observation that the first L3 layer behaves more like a lookup while the second aggregates more suggests a testable hypothesis: stacking multiple L3 layers may yield diminishing returns or require different allocations per depth.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the dense-vs-L3 comparison is real: three model sizes, consistent perplexity gains, and ablations on allocation, layer count, and placement all hang together. Second, the headline claim that L3 'strongly outperforms iso-sparse MoEs' is not supported by the evidence as presented. The MoE baselines look undertuned, and without them the paper's main novel contribution — static token-keyed lookup sparsity — is solid but the comparison to the dominant sparse architecture is not.

What's new: The L3 layer is a clean idea. It generalizes the embedding table into decoder layers with static token-ID routing and context-dependent attention aggregation, and the authors show you can offload and prefetch parameters because routing is known in advance. The LZW-based allocation (Algorithm 1) is a smart twist: treat allocation as a compression problem and allocate embeddings by codeword frequency, with caps and floors. The weight-tying result is a nice bonus — tying K and V costs essentially nothing and halves the parameter overhead. The block-diagonal training implementation is plausible, though the appendix code is sketchy and contains what look like indexing errors (e.g., the double `fw` application in the forward pass). They also do a tuned-lens analysis that shows KL drops at L3 layer positions, which is a good mechanistic check.

Soft spots, in order of severity:

1. MoE baselines. One MoE configuration per sparsity level, no seeds, no sweeps. The 1.5B MoE 'equivalent to one L3 layer' sits *below* the dense baseline after 20B tokens — that is not typical for a well-tuned MoE at this scale. Either the load-balancing weight, router z-loss, or expert count is off, or there's an implementation issue. The paper's own caveat that MoEs 'may require certain token budgets' doesn't explain a gap that large. So the abstract's claim against iso-sparse MoEs is not established.

2. Engrams concurrency. They cite Cheng et al. and say L3 scales similarly, but don't run the comparison. Given how close the mechanisms are, that leaves the novelty claim fuzzy. At minimum, they need to release code and let the community compare.

3. Reproducibility. No error bars, no seeds, no code release (promised later), and the LZW allocation is run on a 1GB slice of the same corpus used for training and eval. The last point isn't circular in a formal sense — the allocation is not fit to perplexity — but it does mean the 'doubled gap' benefit is measured on the same distribution that produced the allocation.

The dense-vs-L3 result stands, though. If the MoE comparison is fixed or removed, this is a useful contribution to the sparse-architecture design space. It deserves a serious referee, but I'd ask the authors to address the baseline issue and release artifacts before accepting.

Bottom line: send it to review, but be explicit that the MoE claim is conditional.

Referee Report

3 major / 6 minor

Summary. The paper proposes L3, a sparse layer that generalizes the tokenizer embedding table into decoder layers. For each token ID, an L3 layer stores a set of key/value embeddings, and the hidden state attends to the active embeddings for that token. Because routing is static and token-based, active parameters are known before the layer executes, enabling block-diagonal attention-style kernels, CPU offloading, and prefetching. An LZW-based allocation algorithm (Algorithm 1) assigns embedding budgets to tokens according to corpus statistics. The authors pretrain Llama-style transformers with 800M/1.5B/2.6B active parameters and report that adding L3 layers improves perplexity and downstream accuracy relative to dense models and to iso-sparse top-k MoE baselines. They also present ablations of layer count, allocation, layer placement, and weight tying, plus a tuned-lens analysis.

Significance. If the empirical claims hold, L3 is a genuinely new axis of sparsity that is substantially more systems-friendly than MoEs: static routing removes the need for load-balancing losses, allows prefetching, and enables offloading with minimal overhead. The dense-vs-L3 evidence is internally consistent and unusually well ablated (Figs. 6, 7C/L/R, 9; Table 1), and the tuned-lens KL drops at L3 positions are a nice mechanistic corroboration. The LZW allocation idea is original, and the ablation against uniform allocation supports its importance. The paper includes pseudocode and a PyTorch implementation in the appendix, aiding reproducibility. The main weakness is the MoE comparison, which rests on a single configuration per sparsity level and is not yet sufficient to support the abstract's headline claim that L3 'strongly outperforms ... iso-sparse MoEs.'

major comments (3)

[§4.1, Table 5, Fig. 8] The MoE comparison is not yet sufficient to support the abstract's claim that L3 'strongly outperforms ... iso-sparse MoEs.' Each sparsity level uses a single MoE configuration, with no seeds, error bars, or sensitivity analysis for the load-balancing weight (0.01), router z-loss (0.001), expert count (15–44), or active experts (4–6). The 1.5B '1 L3 MoE equiv' model underperforms the dense baseline after 20B tokens, which is atypical for standard top-k MoEs at this scale and suggests tuning/configuration issues rather than a fundamental limitation. The paper states it follows OLMoE recommendations, but Table 5 does not use the OLMoE-1B-7B configuration (64 experts, 8 active). Table 5 also omits total parameters, so iso-sparsity cannot be verified. Please strengthen the baselines (e.g., a known-good configuration plus a small sweep) or soften the MoE claim.
[§4.1, Tables 3–4, A.4] The label 'iso-FLOP' is not exact. For the 1.5B and 2.6B classes, the L3 models keep the same decoder depth and hidden/intermediate sizes as the dense baselines and add L3 layers; using the FLOP counts in A.4 this is roughly +1–2.5% FLOPs rather than equal. Only the 800M class drops one decoder layer when adding L3. Also, 'active params/token' in Table 1 rounds the L3 active matrices into the same value as the dense model; exact counts would make the comparison transparent. This does not invalidate the dense-vs-L3 result, but the comparison should be stated accurately.
[§A.3.2, Table 1] Downstream evaluations report raw 'accuracy' rather than 'acc_norm'. For ARC-E, HellaSwag, PIQA, and WinoGrande, acc_norm is the standard LM-Eval metric and avoids length/verbosity bias. Several reported gains are small (e.g., PIQA 71.22→71.71 and WinoGrande 57.30→58.80 for the 2.6B model), so the downstream claim should be verified with acc_norm or with a justification for raw accuracy.

minor comments (6)

[General] Typos include 'Futhermore' (§4.1), 'a setof' (§3), and 'ae three example generations' (§A.5).
[§A.4] The W_mix FLOPs formula appears dimensionally wrong: the code has W_mix = Linear(d_up + h, h), so FLOPs should be 2*h*(d_up + h), not 2*d_emb*(d_emb + d_up). Also, d_out is introduced in §3 but unused in A.4.
[Algorithm 1] The notation 'Count AC = 1^|τ|' is ambiguous. Also state explicitly that the loop terminates because the chosen v satisfies v < k·|τ|; otherwise the cap could prevent reaching the target allocation.
[Table 5/Fig. 8] Give exact sparsity ratios and total parameters for the MoE baselines so that 'iso-sparse' can be checked from the table.
[§3.1] The phrase 'information-theoretic' is a stretch for a frequency-based LZW heuristic without an optimality or code-length argument. Suggest softening to 'compression-inspired.'
[§A.5] The third example generation is from a prompt in the training set; this demonstrates memorization of a training passage rather than open-ended generalization. Label it as such.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: L3's central empirical claims rest on direct measurements, not on fitted predictions or self-citation chains.

full rationale

The paper's derivation chain is: define the L3 layer, choose embedding allocations via Algorithm 1, train dense/L3/MoE models under iso-FLOP conditions, and directly measure perplexity and downstream accuracy. No load-bearing step reduces to its own inputs. The LZW allocation is computed from token frequencies in a corpus slice and is not fit to the evaluation metric; the ablation against uniform allocation is a direct comparison. The dense power-law fit P=aN^b+c is used descriptively to express scaling and is not used to generate the L3 perplexity numbers, which are measured. The MoE comparison is empirical, and any concern about baseline tuning or representativeness is a correctness risk, not circularity. The paper's self-citations appear only in related-work discussions of quantization and sparsity and are not load-bearing for the L3 claims. No quoted reduction of the form 'prediction equals fitted input by construction' or 'choice forced by a self-cited uniqueness theorem' can be exhibited. Therefore the appropriate finding is no circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

L3 introduces no invented physical entities: the token-ID-keyed embedding tables are ordinary learned parameters. The ledger is dominated by hand-chosen hyperparameters (v, k, placement, MoE configs), the data-derived LZW allocation, and the fitted dense power-law used for the scaling claim. The LZW allocation is a preprocessing step on the training corpus, not an entity. The main domain assumptions are the compression-duality heuristic and the representativeness of the MoE baselines.

free parameters (6)

total embedding budget v = 710000
Chosen per experiment; ablated in Fig. 7(L) (355K/710K/1420K variants perform similarly), so it is a tuned design choice rather than an output of the method.
per-token embedding cap k = 512
Caps worst-case active parameters; ablation (Fig. 7C) shows uncapped LZW performs better (>20K embeddings for the most frequent token), so k=512 trades quality for access guarantees.
L3 layer placement = after decoder layers 4 and 16 (2.6B); layer 4 for 800M/1.5B single-layer runs
Placement is ablated in Fig. 9 and changes final PPL by ~0.6 across the tested range; chosen by hand, not derived.
LZW allocation training corpus = 1GB slice of FineWeb-Edu
The allocation (Algorithm 1) is fit to a 1GB slice of the same corpus used for pretraining and evaluation; the 'doubled perplexity gap' of LZW vs. uniform is measured on that distribution.
dense power-law (a, b, c) = fit to 3 dense models
P = aN^b + c fitted to 800M/1.5B/2.6B dense points; extrapolated to claim L3's total parameters correspond to a larger 'equivalent dense model' (Fig. 1, §4.1).
MoE expert counts and active experts = 22/4, 44/4, 15/6, 20/4 depending on config
Hand-set to match target sparsity; the anomalous result that a 1.5B MoE underperforms dense after 20B tokens suggests these hyperparameters are load-bearing for the MoE comparison.

assumptions (5)

domain assumption Suffix-set selection for embedding allocation is effectively dual to lossless text compression, so LZW is a valid allocation route.
§3.1: 'This is effectively a dual problem to lossless text compression...' — asserted without proof; supported only by the 800M ablation (Fig. 7C).
domain assumption Active-parameter count is the fair quality-normalization axis for cross-architecture comparison.
Throughout §4 the comparison is 'iso-FLOP, iso-sparse'; L3 adds ~22M FLOPs/token/layer (A.4) and ~4.4B total extra parameters at 2.6B, which the framing treats as nearly free. Whether caching embeddings substitutes for computation is the premise being tested.
domain assumption The OLMoE-style recipe (softmax router + load-balancing loss + router z-loss) yields competitive MoE baselines at 10-30B tokens.
§4.1/Figure 8; the anomalous result that a 1.5B MoE underperforms the dense model suggests this premise may not hold at these scales, weakening the L3-vs-MoE conclusion.
ad hoc to paper Tuned-lens sharp KL drops indicate information caching by L3 layers.
§4.3/Figure 10; the probe is trained for 500 epochs on a devset and the interpretation ('caching information') is post hoc, but it is ancillary to the central claim.
domain assumption Static routing makes parameter prefetch fully maskable by pre-L3 compute (hardware behavior).
§3.2/Figure 4/Table 2; the no-overhead CPU-offload claim assumes the ≤1MB of L3 parameters per token can be transferred during pre-L3 decoder compute; measured at batch size 1 with 4 decoder layers needed to fully mask.

how reviews work

0 comments

Cite this review

Pith. "Pith review of L$^3$: Large Lookup Layers." pith.science (2026). https://pith.science/paper/FKRBCYH4

@misc{pith2026260121461,
  author       = {Pith},
  title        = {Pith review of: L$^3$: Large Lookup Layers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FKRBCYH4}},
  note         = {Machine review of arXiv:2601.21461}
}

abstract

Modern sparse language models typically achieve sparsity through Mixture-of-Experts (MoE) layers, which dynamically route tokens to dense MLP "experts." However, dynamic hard routing has a number of drawbacks, such as potentially poor hardware efficiency and needing auxiliary losses for stable training. In contrast, the tokenizer embedding table, which is natively sparse, largely avoids these issues by selecting a single embedding per token at the cost of not having contextual information. In this work, we introduce the Large Lookup Layer (L$^3$), which generalizes embedding tables to model decoder layers as a means of further scaling sparsity. L$^3$ layers use static token-based routing to aggregate a set of learned embeddings per token in a context-dependent way, allowing the model to efficiently balance memory and compute by caching information in embeddings. L$^3$ has two main components: (1) a systems-friendly architecture that allows for fast training and CPU-offloaded inference with no overhead, and (2) an information-theoretic embedding allocation algorithm that effectively balances speed and quality. We empirically test L$^3$ by training transformers with up to 2.6B active parameters and find that L$^3$ strongly outperforms both dense models and iso-sparse MoEs in both language modeling and downstream tasks.

Figures

Figures reproduced from arXiv: 2601.21461 by the authors.

**Figure 1.** L 3 scaling. Adding sparsity with L3 layers significantly improves performance over iso-FLOP dense models (⋆) and isoFLOP, iso-sparse MoEs (×). “experts.” In an MoE, the router performs context-dependent routing of each token to a subset of the experts; this subset is “activated” for that particular token. While MoEs perform well, context-dependent routing is not very systems-friendly (Zoph et al., 2022). Routers m… view at source ↗

**Figure 2.** The basic architecture of a L3 layer for a single token. The token ID t is used to select embeddings Kt, Vt. Embeddings are aggregated with attention from the token hidden state x. The attention output is combined with the residual stream (hidden state x) to produce the L3 output. Embeddings are assigned to token IDs with an information-theoretic LZW-based algorithm. et al., 2017; Dai et al., 2024; Muennighoff et al… view at source ↗

**Figure 3.** Efficient training with L3 . Since L3 only performs channel mixing, all tokens in a batch can be sorted to form a block-diagonal “attention mask” for L3 embedding aggregation. This allows L3 to use fast kernels such as MegaBlocks or FlexAttention. The only overhead is sorting tokens, which can be done ahead of time, and inverting the sort, which is trivial. GPU Sample Token t Pre-L3 Model Compute L 3 Post-L3 Model C… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Fast offloaded inference with L3 . Since the set of needed L 3 parameters is known when a token is generated, L3 parameters can be offloaded and prefetched during pre-L3 compute. 0 100 200 300 400 500 Embedding Count 10 1 10 2 10 3 10 4 10 5 Tokens [PITH_FULL_IMAGE:fi…

**Figure 5.** Figure 5: Histogram of embedding count for 710K embeddings, a 180K vocab BPE tokenizer, and Algorithm 1 with k = 512. “Common” tokens such as “then” get 512 embeddings while “rare” tokens such as “orm” get 1 embedding. then outputs L 3 (x, t; Wmix, Wup, K, V ) ∈ R dout = Wmix …

**Figure 6.** Figure 6: Training perplexity on FineWeb-Edu, dense and L3 transformers. L3 layers significantly improve performance [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: (L) Ablation on the number of L3 layers and parameters per L3 layer for a target sparsity rate. (C) Ablations on LZW allocation. Uncapped LZW performs the best but has poor access guarantees, with the most “common” token getting over 20K embeddings out of 710K. Using u…

**Figure 10.** Figure 10: Tuned lens on 2.6B L3 and dense models. L3 layers induce sharp drops in KL across decoder layers, indicating that the model is caching information in L3 layers [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]

**Figure 8.** Figure 8: Perplexity gap vs. dense model for L3 models and iso-FLOP and iso-sparse MoEs. At all tested sparsity levels, L3 s outperform equivalent MoEs. 2 4 6 8 10 12 14 16 After Decoder Layer 27.6 27.8 28.0 28.2 Final Training Perplexity L 3 Placement [PITH_FULL_IMAGE:figures/…

**Figure 9.** Figure 9: Final training perplexity of a 800M class model with a single L3 layer with 355K embeddings placed after different decoder layers. Models trained for 1B tokens. the expert intermediate dimension to hit a specific total sparsity. Exact details are in the Appendix. At a…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 linked inside Pith

[2020]

train” split of the “Sample-350BT

as a basis. We use the Llama RMS Norm with ϵ= 10 −5, and the RoPE scaling and θ configuration from Llama 3.2 1B. We train models with the AdamW optimizer (Loshchilov & Hutter, 2019) with the default PyTorch ϵ= 10 −8 and a cosine schedule learning rate (Loshchilov & Hutter, 2017). We use the PyTorch implementations of FSDP sharding with BF16 mixed precisio...

2019
[2024]

Tseng, A., Sun, Q., Hou, D., and Sa, C

URL https://openreview.net/forum? id=9BrydUVcoe. Tseng, A., Sun, Q., Hou, D., and Sa, C. D. Qtip: Quanti- zation with trellises and incoherence processing, 2025a. URLhttps://arxiv.org/abs/2406.11235. Tseng, A., Sun, Z., and Sa, C. D. Model-preserving adaptive rounding, 2025b. URL https://arxiv.org/abs/ 2505.22988. Tseng, A., Yu, T., and Park, Y . Training...

arXiv 2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2020]

train” split of the “Sample-350BT

as a basis. We use the Llama RMS Norm with ϵ= 10 −5, and the RoPE scaling and θ configuration from Llama 3.2 1B. We train models with the AdamW optimizer (Loshchilov & Hutter, 2019) with the default PyTorch ϵ= 10 −8 and a cosine schedule learning rate (Loshchilov & Hutter, 2017). We use the PyTorch implementations of FSDP sharding with BF16 mixed precisio...

2019

[2] [2024]

Tseng, A., Sun, Q., Hou, D., and Sa, C

URL https://openreview.net/forum? id=9BrydUVcoe. Tseng, A., Sun, Q., Hou, D., and Sa, C. D. Qtip: Quanti- zation with trellises and incoherence processing, 2025a. URLhttps://arxiv.org/abs/2406.11235. Tseng, A., Sun, Z., and Sa, C. D. Model-preserving adaptive rounding, 2025b. URL https://arxiv.org/abs/ 2505.22988. Tseng, A., Yu, T., and Park, Y . Training...

arXiv 2024