Tokenizer-Agnostic Engram Module

Hai Leong Chieu; Jia Peng Lim

REVIEW 2 major objections 5 minor 13 references

By hashing raw bytes with a polynomial instead of XOR-ing token hashes, the Engram conditional memory module can reuse its pretrained embeddings across models that use different tokenizers, with no loss in performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:16 UTC pith:JAME3G7R

load-bearing objection Solid, incremental idea with a correct hashing derivation and suggestive transfer results; the missing byte-normalization spec and one missing control keep it from being fully convincing. the 2 major comments →

arxiv 2607.29065 v1 pith:JAME3G7R submitted 2026-07-31 cs.CL

Tokenizer-Agnostic Engram Module

Jia Peng Lim , Hai Leong Chieu This is my paper

classification cs.CL

keywords Engramtokenizer-agnosticpolynomial hashingN-grambyte-level representationconditional memoryhash embeddingscross-tokenizer transfer

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that the Engram module—a conditional memory lookup used in large language models—need not be tied to a single tokenizer. The authors argue that what matters is the underlying byte sequence of an N-gram, not the tokenizer's particular partition, and they replace the original XOR-based token hashing with byte-level polynomial hashing. This makes byte-equivalent token sequences hash to the same embedding index, so embeddings trained with one tokenizer can be reused by a model with a different tokenizer. They show this substitution performs comparably to the original in pretraining, and that cross-tokenizer transfer of frozen Engram embeddings improves downstream benchmarks in a smaller model. Why care: it decouples memory modules from tokenizer choice, letting models of different sizes or vocabularies share pretrained memory weights.

Core claim

The central claim is that replacing the token-level XOR hash with general polynomial hashing over the raw bytes of token sequences yields hash equivalence for byte-equivalent sequences, and that this, together with a joint N-gram embedding space and the inclusion of 1-grams, makes Engram embeddings tokenizer-agnostic. The paper derives a streaming form of polynomial hashing with cached per-token hashes and base exponents, preserving the same asymptotic complexity as the original XOR routine. Experiments on three backbones show comparable pretraining performance to the XOR version, and a transfer experiment—freezing embeddings pretrained with a large-vocabulary reference tokenizer and using t

What carries the argument

The load-bearing mechanism is the general polynomial hash Hash_p(S) = [Σ_{i=0}^{|S|-1} h(s_i)·base^{|S|-i}] mod M, applied to raw bytes rather than tokens. Its streaming form, Hash_p(T(S)) = ((C(t0)·B(|t1|)+C(t1))·B(|t2|)+...+C(t_{k-1})) mod M, with cached per-token hash C(t)=Hash_p(t) and base exponent B(p)=base^p, makes the hash of any token sequence equal to the hash of the concatenated bytes, so different tokenizations of the same byte string collide by construction. This is what converts token-level N-gram lookups into byte-sequence lookups.

Load-bearing premise

The result assumes that two tokenizers encode the same surface string with the same byte sequence once their tokenizer-specific space markers and special tokens are normalized, but the paper asserts this normalization without specifying it, so byte-level hash equivalence across real tokenizers is not demonstrated for raw, unnormalized bytes.

What would settle it

Use two real tokenizers whose space-prefix bytes differ (e.g., one encodes spaces as an underscore and the other as a Unicode marker), tokenize the same sentence, and compute the polynomial hash of each token sequence using the paper's streaming formula without any marker normalization; if the two hashes differ for the same surface string, tokenizer-agnosticism fails unless the unspecified normalization is applied.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Pretrained Engram embeddings can be shared across models with different tokenizers, removing the need to retrain memory modules from scratch.
Model developers can adjust tokenizer vocabulary size to match model scale without abandoning previously trained memory weights.
The joint embedding space across N introduces no observed pretraining penalty, suggesting byte-level information is the useful carrier rather than N-gram identity.
Cross-tokenizer transfer of frozen embeddings yields higher benchmark scores than a same-size reference model, indicating the transferred N-gram lookups are semantically meaningful.
Because the hashing change is a drop-in substitution with the same asymptotic complexity, the tokenizer-agnostic property comes at negligible computational cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the byte-equivalence property holds under a documented normalization of spaces and special tokens, the same polynomial-hashing scheme should extend to multilingual and byte-level tokenizers, potentially making memory modules shareable across scripts and languages.
The paper's framing of N-grams as a sampling mechanism suggests a testable corollary for other N-gram-based components: any lookup keyed by token N-grams could be rebased onto byte sequences to gain the same transferability.
One could stress-test the claim by training Engram embeddings with a byte-level tokenizer (e.g., character/byte BPE) and transferring to a wordpiece-style tokenizer; the paper did not include this extreme case.
The ablation that 1-grams are not the driver implies that the transfer signal lives in multi-token byte sequences; a natural follow-up is to measure how transfer quality scales with the fraction of overlapping byte N-grams between tokenizer pairs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid, incremental idea with a correct hashing derivation and suggestive transfer results; the missing byte-normalization spec and one missing control keep it from being fully convincing.

read the letter

Quick take: this is a legitimate incremental contribution. The core idea — hash raw bytes with polynomial hashing instead of token IDs with XOR so that Engram embeddings can be shared across tokenizers — is clearly stated, and the math in Eqs. 3-8 checks out. The joint N-gram space and the 1-gram inclusion are sensible additions, and the paper does a decent job of explaining why token-level N-grams are not the right unit. The empirical overlap analysis (Table 2) is a nice sanity check that byte-equivalent N-grams actually exist across tokenizers.

What's less solid: the paper never specifies how space markers and special tokens are normalized to bytes. The stress-test note is right that '_This' and 'Ġis' are not byte-equivalent on the raw UTF-8 level, so Eq. 1 only works if there is a canonical mapping. The paper says it 'accounts for' these but doesn't say how. That's a real gap, but it's a spec gap, not a hole in the math. I'd want the normalization defined before trusting the tokenizer-agnosticism claim.

Second, the cross-tokenizer transfer experiment (Table 4) lacks a control with a randomly initialized Engram module of the same size. Without that, it's hard to attribute the gains to the pretrained embeddings being tokenizer-agnostic rather than to extra frozen capacity. The N=1 ablation (Table 5) is informative but doesn't close this gap.

Also worth noting: every configuration is trained once. The differences in Table 3 are small and could be noise. The authors acknowledge the standard errors on benchmarks, but not run-to-run variance.

What the paper does well: it's honest about what's new and what isn't, cites prior work on polynomial hashing, and ships code. The derivation of the streaming hash (Eqs. 4-8) is clean and correctly shows the efficiency argument.

My bottom line: this deserves a serious referee. The central idea is plausible and the math is right; the empirical claims need a bit more discipline (normalization spec, random-init baseline, maybe a second seed). It's a workshop-to-conference quality paper that could be a solid contribution with modest revisions.

Referee Report

2 major / 5 minor

Summary. The paper addresses a limitation of DeepSeek's Engram module, whose token-level XOR N-gram hashing ties the learned embeddings to a specific tokenizer. The authors propose replacing XOR hashing with polynomial hashing over raw bytes, along with a joint embedding space across N-gram orders and the inclusion of 1-grams, so that byte-equivalent token sequences from different tokenizers map to the same embedding index. They prove the polynomial-hash equivalence (Eqs. 3–8), analyze the overlap of byte-equivalent N-grams across three tokenizers on Wikitext, and report pretraining experiments showing comparable performance to the original Engram and a cross-tokenizer transfer experiment.

Significance. If the byte-normalization and experimental-control concerns are resolved, the contribution is valuable: it provides a principled, computationally feasible way to reuse Engram embeddings across tokenizers, which is important for the practical deployment of conditional memory modules. The mathematical derivation is clear and correct, the code is released, and the empirical analysis of byte-equivalent N-gram overlap is informative. The main open risks are whether the implementation actually achieves byte equivalence across real tokenizers and whether the transfer gains are specifically due to the pretrained embeddings rather than other sources of additional capacity.

major comments (2)

[Hashing for Tokenizer-Agnosticism, first paragraph and Eqs. (1)-(2)] The paper claims hash equivalence for byte-equivalent token sequences, but the byte-equivalence relation in Eq. 1 is not established for the actual tokenizers because the canonical byte representation of tokenizer-specific markers is never defined. The text says "we also account for special tokens and space prefixes between different tokenizers," but no concrete mapping is given. For example, in Table 1, the space-prefix tokens '_This' (Mistral) and 'Ġis' (SmolLM2) have different raw UTF-8 bytes (0x5F vs. 0xC4 0xA0), and special tokens like '<s>' and '<|endoftext|>' are not surface-form bytes. Unless the implementation converts these markers to a common byte (e.g., a single ASCII space for all space prefixes, and a fixed byte or empty sequence for special tokens), Eq. 1 fails and the subsequent polynomial hash equivalence does not hold. Since the entire tokenizer-agnosticism claim rests
[Training Tokenizer-agnostic Engrams (Table 4)] The cross-tokenizer transfer experiment lacks a control with randomly initialized frozen Engram embeddings. The observed improvement of ModelB + Engram over the reference model could in principle be driven by the additional trainable parameters (the ~15M non-embedding parameters) or by the capacity of the frozen embedding table, rather than by the fact that the embeddings were pretrained on a different tokenizer and accessed byte-equivalent N-grams. To substantiate the claim that "cross-tokenizer lookups on pretrained Engram embeddings are successful," the authors should train/test a variant of ModelB with the same architecture and a randomly initialized frozen Engram table of the same size. Without such a control, the improvement cannot be uniquely attributed to tokenizer-agnosticism.

minor comments (5)

[Table 2 and Figure 2] The captions and axis labels are ambiguous about whether the reported percentages are within the same N, across different N, and how "found" is defined (byte-equivalence vs. exact token equality). Please clarify the computation and the aggregation across reference N-grams.
[Introduction, contribution 1] Typo: "Tokenizer Agnosticsm" should be "Tokenizer Agnosticism."
[Experiments, comparing XOR and Poly hashing] Each configuration in Table 3 is trained once. The conclusion that polynomial hashing "does not negatively affect training" would be stronger with multiple random seeds or at least a statement acknowledging that the observed differences could include training noise. The reported standard error (≤0.015) appears to be the benchmark evaluation error, not the across-seed variance.
[Experiments, 1-gram scaling] The 0.1 scaling factor for 1-gram embeddings is stated in a parenthetical but is not listed among the primary hyperparameters. Please specify it clearly in the experiment setup and, if possible, provide a brief sensitivity note.
[Polynomial-hashing as an Efficient Alternative] The sentence "We use the general polynomial hashing approach for this work. Other specific hashing approaches, such as Bernstein (2005); Degabriele et al. (2024), may be applicable" is somewhat vague. It would help to state whether the byte-hash function h and base are fixed universal parameters or tuned hyperparameters, and how collisions are handled.

Circularity Check

0 steps flagged

No circularity: hash-equivalence proof is a direct algebraic identity; empirical transfer is benchmarked externally.

full rationale

The central derivation is self-contained. The paper defines tokenizer agnosticism as hash equivalence for byte-equivalent token sequences (Eq. 2), then proves that the proposed polynomial hashing has this property: Eq. 3 (byte-level polynomial hash) is algebraically identical to Eqs. 7–8 (cached token hashes weighted by byte lengths), with the example in Eq. 9 demonstrating Hash_p(S) = Hash_p(T(S)). This is a mathematical identity, not a fit or a renamed empirical result. The empirical sections are external benchmarks comparing XOR vs polynomial Engram variants (Table 3), cross-tokenizer transfer with frozen pretrained embeddings (Table 4), and an N=1 ablation (Table 5); none of these use the target result as a fitted input. No load-bearing self-citation or uniqueness import is present. The only substantive concern is that the normalization of tokenizer-specific markers (space prefixes, special tokens) into a common byte representation is asserted without specification; that is a potential correctness/verifiability gap in applying Eq. 1 to real tokenizers, but it does not make the derivation circular.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central hash-equivalence claim rests only on standard polynomial-hash math. However, the empirical tokenizer-agnosticism claim depends on unstated normalization and the byte-level information hypothesis, which are domain assumptions.

free parameters (5)

base
Base of the polynomial hash; a hyperparameter. Not fitted to data, but affects collision rate and hash distribution.
M
Engram embedding table size (e.g., 1M or 2M indices). Chosen based on parameter budget, not fitted.
N
Maximum N-gram order (N=3 for from-scratch experiments, N=7 for transfer). Chosen by the experimenters, not predicted by theory.
byte-hash function h
Fixed mapping from bytes to integers used in hashing. Not learned; presumably random. Its distribution is assumed to be sufficiently uniform to avoid systematic hash collisions.
1-gram scaling factor = 0.1
Scaling factor applied to 1-gram embedding values at the start of training to reduce initial impact. Chosen by hand.

axioms (5)

standard math Polynomial hashing with cached token hashes and base powers composes correctly for token sequences (Eqs. 3-8).
Standard property of polynomial hash; used to derive Eq. 7/8.
domain assumption Space prefixes and special tokens can be normalized so that different tokenizers produce identical byte sequences for the same surface text.
Assumed in 'Hashing for Tokenizer-Agnosticism' but not specified in detail.
domain assumption The byte-level hash function h distributes values uniformly enough that collisions are tolerable and can be gated by the context-aware SPDA.
Assumed for the joint embedding lookup; no collision analysis is provided.
domain assumption A joint embedding space across N does not degrade learning relative to disjoint N-specific spaces.
Empirically supported by Table 3, not derived from first principles.
domain assumption The useful information in Engram embeddings lies at the byte level, not tied to specific token N-grams.
This is the paper's central perspective; it is a hypothesis supported by the N-gram overlap analysis and the N=1 ablation, but not independently established.

pith-pipeline@v1.3.0-daily-deepseek · 13606 in / 13442 out tokens · 128761 ms · 2026-08-03T14:16:50.267705+00:00 · methodology

0 comments

read the original abstract

Deepseek's Engram, a conditional memory module, was introduced to trade-off storage versus reasoning in large language models. However, the module relies on token-level $N$-gram hashing for Engram embedding lookup, introducing a tight coupling to the tokenizer used: a model with a different tokenizer would have to train its own Engram embeddings from scratch. To improve the reusability of Engram embeddings, we propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers. Instead of modelling disjoint $N$-gram spaces, we treat $N$-gram as a method to sample potentially useful byte sequences, from all possible byte sequences across tokens. We replace the XOR-based hashing with the general polynomial hashing with a joint embedding space across $N$. This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance and achieves tokenizer-agnosticism: hash equivalence for byte-equivalent token sequences.

Figures

Figures reproduced from arXiv: 2607.29065 by Hai Leong Chieu, Jia Peng Lim.

**Figure 1.** Figure 1: Visualization of the algorithmic and architectural changes between the original approach and our proposed approach. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Figure 2: Percentages of unique N-grams sampled on wikitext val using different target tokenizers (Mistral and SmolLM2) that exists when using reference tokenizer (cl100k_base). N-gram pairs, on the diagonal, might be partitioned differently. N Target 1 2 3 4 5 6 7 Mistral 0.80 0.81 0.80 0.78 0.75 0.74 0.70 SmolLM2 0.92 0.91 0.89 0.86 0.86 0.85 0.80 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Plots of difference in training loss between Engram [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

13 extracted references · 7 linked inside Pith

[4]

JohannesWelbl,M.G.,NelsonF.Liu.2017

Mistral 7B.arXiv preprint arXiv:2310.06825. JohannesWelbl,M.G.,NelsonF.Liu.2017. Crowdsourcing Multiple Choice Science Questions.arXiv:1707.06209v1. Kudo, T

Pith/arXiv arXiv 2017
[9]

InThe Fourteenth International Con- ference on Learning Representations

STEM: SCALING TRANSFORMERS WITH EMBED- DING MODULES. InThe Fourteenth International Con- ference on Learning Representations. Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019.WinoGrande:AnAdversarialWinogradSchemaChal- lenge at Scale.arXiv preprint arXiv:1907.10641. Schmidt, C. W.; Reddy, V.; Tanner, C.; and Pinter, Y

Pith/arXiv arXiv 2019
[10]

InChe,W.; Nabende, J.; Shutova, E.; and Pilehvar, M

Large Vo- cabularySizeImprovesLargeLanguageModels. InChe,W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds.,Findings oftheAssociationforComputationalLinguistics:ACL2025, 1015–1026.Vienna,Austria:AssociationforComputational Linguistics. ISBN 979-8-89176-256-5. Tao,C.;Liu,Q.;Dou,L.;Muennighoff,N.;Wan,Z.;Luo,P.; Lin,M.;andWong,N.2024. ScalingLawswithVoc...

2024
[11]

Tseng,A.;andSa,C.D.2026

Llama 2: Open Foundation and Fine-Tuned Chat Models. Tseng,A.;andSa,C.D.2026. $L^3$:LargeLookupLayers. InForty-third International Conference on Machine Learn- ing. Vaswani,A.;Shazeer,N.;Parmar,N.;Uszkoreit,J.;Jones,L.; Gomez,A.N.;Kaiser,L.;andPolosukhin,I.2017. Attention Is All You Need.arXiv preprint arXiv:1706.03762. Yang,A.;Yang,B.;Hui,B.;Zheng,B.;Yu,...

Pith/arXiv arXiv 2026
[13]

Zheng,Y.;Xia,G.;Wang,X.;andRen,L.2026

HellaSwag: Can a Machine Really Finish Your Sentence? InProceedingsofthe57thAnnualMeetingofthe Association for Computational Linguistics. Zheng,Y.;Xia,G.;Wang,X.;andRen,L.2026. Lngram:N- gram Conditional Memory in Latent Space.arXiv preprint arXiv:2605.24869

Pith/arXiv arXiv 2026
[2005]

Springer

The Poly1305-AES Message- AuthenticationCode.InGilbert,H.;andHandschuh,H.,eds., FastSoftwareEncryption:12thInternationalWorkshop,FSE 2005,Paris,France,February21–23,2005,RevisedSelected Papers,volume3557ofLectureNotesinComputerScience, 32–49. Springer. ISBN 3-540-26541-4. Bhattacharyya,S.;Nath,K.;andSarkar,P.2025. Polynomial hashing over prime order field...

2005
[2016]

Merrill,W.;Smith,N.A.;andElazar,Y.2024

Pointer Sentinel Mixture Models.arXiv:1609.07843. Merrill,W.;Smith,N.A.;andElazar,Y.2024. Evaluatingn- Gram Novelty of Language Models Using Rusty-DAWG. In Al-Onaizan,Y.;Bansal,M.;andChen,Y.-N.,eds.,Proceed- ingsofthe2024ConferenceonEmpiricalMethodsinNatural Language Processing, 14459–14473. Miami, Florida, USA: Association for Computational Linguistics. ...

Pith/arXiv arXiv 2024
[2018]

In Blanco, E.; and Lu, W., eds.,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, 66–71

SentencePiece: A sim- ple and language independent subword tokenizer and deto- kenizer for Neural Text Processing. In Blanco, E.; and Lu, W., eds.,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, 66–71. Brussels, Belgium: Association for Com- putational Linguistics. Levesque, H. J.; Davis, E....

2018
[2019]

BoolQ: Exploring the Surpris- ing Difficulty of Natural Yes/No Questions. InNAACL. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick,C.;andTafjord,O.2018. ThinkyouhaveSolved Question Answering? Try ARC, the AI2 Reasoning Chal- lenge.arXiv:1803.05457v1. Degabriele, J. P.; Gilcher, J.; Govinden, J.; and Paterson, K. G

Pith/arXiv arXiv 2018
[2023]

Allal, L

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Check- points.arXiv preprint arXiv:2305.13245. Allal, L. B.; Lozhkov, A.; Bakouch, E.; Blázquez, G. M.; Penedo,G.;Tunstall,L.;Marafioti,A.;Kydlíček,H.;Lajarín, A.P.;Srivastav,V.;Lochner,J.;Fahlgren,C.;Nguyen,X.-S.; Fourrier, C.; Burtenshaw, B.; Larcher, H.; Zhao, H.; Zakka, C.;Morlon...

Pith/arXiv arXiv 2025
[2024]

Yang,S.; Kautz,J.; andHatamizadeh, A.2025

Qwen2 Technical Report. Yang,S.; Kautz,J.; andHatamizadeh, A.2025. GatedDelta Networks:ImprovingMamba2withDeltaRule. InTheThir- teenth International Conference on Learning Representa- tions. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y

2025
[2025]

InTokenization Workshop

SuperBPE: Space Travel for Language Models. InTokenization Workshop. Liu, J.; Min, S.; Zettlemoyer, L.; Choi, Y.; and Hajishirzi, H.2024. Infini-gram:ScalingUnboundedn-gramLanguage Models to a Trillion Tokens. InFirst Conference on Lan- guage Modeling. Loshchilov,I.;andHutter,F.2019. DecoupledWeightDecay Regularization. InInternational Conference on Learn...

2024
[2026]

Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever,I.2019

Qwen3.5: Towards Native Multimodal Agents. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever,I.2019. LanguageModelsareUnsupervisedMul- titask Learners. Roemmele,M.;Bejan,C.A.;andGordon,A.S.2011.Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In2011 AAAI Spring Symposium Series. Sadhukhan, R.; Cao, S.; Don...

2019

[1] [4]

JohannesWelbl,M.G.,NelsonF.Liu.2017

Mistral 7B.arXiv preprint arXiv:2310.06825. JohannesWelbl,M.G.,NelsonF.Liu.2017. Crowdsourcing Multiple Choice Science Questions.arXiv:1707.06209v1. Kudo, T

Pith/arXiv arXiv 2017

[2] [9]

InThe Fourteenth International Con- ference on Learning Representations

STEM: SCALING TRANSFORMERS WITH EMBED- DING MODULES. InThe Fourteenth International Con- ference on Learning Representations. Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2019.WinoGrande:AnAdversarialWinogradSchemaChal- lenge at Scale.arXiv preprint arXiv:1907.10641. Schmidt, C. W.; Reddy, V.; Tanner, C.; and Pinter, Y

Pith/arXiv arXiv 2019

[3] [10]

InChe,W.; Nabende, J.; Shutova, E.; and Pilehvar, M

Large Vo- cabularySizeImprovesLargeLanguageModels. InChe,W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds.,Findings oftheAssociationforComputationalLinguistics:ACL2025, 1015–1026.Vienna,Austria:AssociationforComputational Linguistics. ISBN 979-8-89176-256-5. Tao,C.;Liu,Q.;Dou,L.;Muennighoff,N.;Wan,Z.;Luo,P.; Lin,M.;andWong,N.2024. ScalingLawswithVoc...

2024

[4] [11]

Tseng,A.;andSa,C.D.2026

Llama 2: Open Foundation and Fine-Tuned Chat Models. Tseng,A.;andSa,C.D.2026. $L^3$:LargeLookupLayers. InForty-third International Conference on Machine Learn- ing. Vaswani,A.;Shazeer,N.;Parmar,N.;Uszkoreit,J.;Jones,L.; Gomez,A.N.;Kaiser,L.;andPolosukhin,I.2017. Attention Is All You Need.arXiv preprint arXiv:1706.03762. Yang,A.;Yang,B.;Hui,B.;Zheng,B.;Yu,...

Pith/arXiv arXiv 2026

[5] [13]

Zheng,Y.;Xia,G.;Wang,X.;andRen,L.2026

HellaSwag: Can a Machine Really Finish Your Sentence? InProceedingsofthe57thAnnualMeetingofthe Association for Computational Linguistics. Zheng,Y.;Xia,G.;Wang,X.;andRen,L.2026. Lngram:N- gram Conditional Memory in Latent Space.arXiv preprint arXiv:2605.24869

Pith/arXiv arXiv 2026

[6] [2005]

Springer

The Poly1305-AES Message- AuthenticationCode.InGilbert,H.;andHandschuh,H.,eds., FastSoftwareEncryption:12thInternationalWorkshop,FSE 2005,Paris,France,February21–23,2005,RevisedSelected Papers,volume3557ofLectureNotesinComputerScience, 32–49. Springer. ISBN 3-540-26541-4. Bhattacharyya,S.;Nath,K.;andSarkar,P.2025. Polynomial hashing over prime order field...

2005

[7] [2016]

Merrill,W.;Smith,N.A.;andElazar,Y.2024

Pointer Sentinel Mixture Models.arXiv:1609.07843. Merrill,W.;Smith,N.A.;andElazar,Y.2024. Evaluatingn- Gram Novelty of Language Models Using Rusty-DAWG. In Al-Onaizan,Y.;Bansal,M.;andChen,Y.-N.,eds.,Proceed- ingsofthe2024ConferenceonEmpiricalMethodsinNatural Language Processing, 14459–14473. Miami, Florida, USA: Association for Computational Linguistics. ...

Pith/arXiv arXiv 2024

[8] [2018]

In Blanco, E.; and Lu, W., eds.,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, 66–71

SentencePiece: A sim- ple and language independent subword tokenizer and deto- kenizer for Neural Text Processing. In Blanco, E.; and Lu, W., eds.,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, 66–71. Brussels, Belgium: Association for Com- putational Linguistics. Levesque, H. J.; Davis, E....

2018

[9] [2019]

BoolQ: Exploring the Surpris- ing Difficulty of Natural Yes/No Questions. InNAACL. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick,C.;andTafjord,O.2018. ThinkyouhaveSolved Question Answering? Try ARC, the AI2 Reasoning Chal- lenge.arXiv:1803.05457v1. Degabriele, J. P.; Gilcher, J.; Govinden, J.; and Paterson, K. G

Pith/arXiv arXiv 2018

[10] [2023]

Allal, L

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Check- points.arXiv preprint arXiv:2305.13245. Allal, L. B.; Lozhkov, A.; Bakouch, E.; Blázquez, G. M.; Penedo,G.;Tunstall,L.;Marafioti,A.;Kydlíček,H.;Lajarín, A.P.;Srivastav,V.;Lochner,J.;Fahlgren,C.;Nguyen,X.-S.; Fourrier, C.; Burtenshaw, B.; Larcher, H.; Zhao, H.; Zakka, C.;Morlon...

Pith/arXiv arXiv 2025

[11] [2024]

Yang,S.; Kautz,J.; andHatamizadeh, A.2025

Qwen2 Technical Report. Yang,S.; Kautz,J.; andHatamizadeh, A.2025. GatedDelta Networks:ImprovingMamba2withDeltaRule. InTheThir- teenth International Conference on Learning Representa- tions. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y

2025

[12] [2025]

InTokenization Workshop

SuperBPE: Space Travel for Language Models. InTokenization Workshop. Liu, J.; Min, S.; Zettlemoyer, L.; Choi, Y.; and Hajishirzi, H.2024. Infini-gram:ScalingUnboundedn-gramLanguage Models to a Trillion Tokens. InFirst Conference on Lan- guage Modeling. Loshchilov,I.;andHutter,F.2019. DecoupledWeightDecay Regularization. InInternational Conference on Learn...

2024

[13] [2026]

Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever,I.2019

Qwen3.5: Towards Native Multimodal Agents. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever,I.2019. LanguageModelsareUnsupervisedMul- titask Learners. Roemmele,M.;Bejan,C.A.;andGordon,A.S.2011.Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In2011 AAAI Spring Symposium Series. Sadhukhan, R.; Cao, S.; Don...

2019