Parameter-Efficient Transformer Embeddings

Henry Ndubuaku; Mouad Talhi

REVIEW 4 major objections 7 minor 21 references

Parameter-Efficient Transformer Embeddings

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a deterministic Fourier expansion of token IDs, followed by a small shared MLP, can replace the learned embedding table of a transformer without sacrificing task accuracy, while using a fraction of the parameters…

desk verdict A small, honest proof-of-concept that deterministic Fourier token embeddings can rival learned embeddings at toy scale; the mechanism is untested and the evidence is thin, but it deserves a referee. read the letter →

arxiv 2505.02266 v1 pith:OFRIDK3X submitted 2025-05-04 cs.CL cs.AIcs.LG

Henry Ndubuaku , Mouad Talhi This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords parameter-efficienttransformersFourierembeddingstokenbyte-pairencodingnaturallanguageinferencecontrastivelearningsentencesimilarityembeddingcompression

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Transformers normally store a learned vector for every token, and that lookup table is usually the largest parameter block in the model. This paper proposes generating each token's embedding deterministically from its token ID, using a fixed Fourier expansion of a normalized ID followed by a small shared MLP, and presents evidence that this substitution matches a standard learned-embedding transformer on sentence-pair tasks while using a fraction of the parameters and less training time. The experiments train on SNLI and MNLI with a contrastive loss and evaluate zero-shot on STS-B; the smallest fine-tuned Fourier models outperform several officially reported tiny BERT variants. If the claim holds, embedding memory no longer needs to scale with vocabulary size, which would matter most for large language models.

What carries the argument

The central object is the Fourier expansion of normalized token IDs: each token ID $p$ is normalized to $x = 2p/(V-1) - 1$, and component $i$ of the base embedding is $T_i(p) = \sin((\lfloor i/2 \rfloor + 1)\pi x)$ for even $i$ and $\cos((\lfloor i/2 \rfloor + 1)\pi x)$ for odd $i$, with final embedding $E(p) = \mathrm{MLP}(T(p)) + T(p)$. Low-order terms capture broad trends across the ID order, high-order terms separate nearby IDs, and the shared MLP is the learnable residual that separates tokens with near-colliding base representations. This turns the embedding layer from a stored $V \times d$ matrix into a small learned function of a scalar.

What would settle it

Shuffle the token IDs randomly before training, holding the vocabulary, model, and data fixed; if a Fourier-embedding transformer trained on shuffled IDs performs about the same as the unshuffled version, the ordering assumption is not doing the work, and if performance drops sharply, the frequency ordering is load-bearing.

Watch

Extended reading notes

Core claim

The paper's central claim is that a parameter-free, deterministic Fourier basis can serve as the token-embedding layer of a transformer, as long as a lightweight shared MLP and a residual connection are added to refine the fixed features. Concretely, each token ID $p$ is mapped to $x \in [-1,1]$, its Fourier expansion up to the embedding dimension forms $T(p)$, and the final embedding is $E(p) = \mathrm{MLP}(T(p)) + T(p)$. On the configurations tested, transformers using Fourier embeddings reached STS-B scores within roughly a point of standard transformers while using about one-eighth to one-quarter of the parameters, and training was faster; the paper's largest PETE model after fine-tuning reached a Spearman correlation of 81.9, above the official BERT-Tiny and BERT-Mini reports. The authors read this as support for the hypothesis that the frequency ordering of byte-pair-encoding token IDs carries weak but exploitable structure, which the fixed basis captures and the MLP corrects.

Load-bearing premise

The method rests on the premise that the frequency-based ordering of BPE token IDs correlates with useful embedding structure; the paper itself warns that for very large vocabularies the normalized ID space becomes densely packed, so distinct tokens' base Fourier embeddings nearly collide and the MLP must separate a dense field of near-identical inputs, which has not been demonstrated at scale.

Editorial extensions

If this is right

Embedding parameter count becomes independent of vocabulary size; adding tokens to the vocabulary no longer adds parameters to the embedding layer.
Because the embedding function is fixed and smooth, models can be trained without dropout, and custom kernels that fuse normalization and Fourier expansion cut training time.
Freed parameter budget can be moved from the embedding table into attention depth or width, which is where the paper observes the best returns.
At equal total parameter counts, the Fourier-embedding transformers matched or exceeded the traditional transformer on STS-B, suggesting learned tables waste capacity at small scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the ordering assumption holds at scale, this design could make embedding layers essentially free in web-scale models, where vocabularies of hundreds of thousands of tokens dominate memory.
The smooth ID-to-vector map may give interpretability: probing how the final embedding evolves with token ID could expose the model's internal frequency-to-semantics organization.
A direct stress test is vocabulary randomization on larger models; if performance collapses, frequency ordering is load-bearing, and if it does not, the MLP alone is doing the work.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The one genuinely new thing is the empirical demonstration that a fixed Fourier expansion of normalized token IDs plus a shared MLP can get within a point or two of learned embeddings on zero-shot STS-B, at a fraction of the parameters. That result in Table 1 is plausible as far as it goes, and the authors are refreshingly honest about the toy scale and resource limits. They also release code, which makes this reproducible rather than just suggestive.

The soft spots are real but not fatal. First, there are no error bars or repeated seeds, so the 1–3 point gaps could easily be noise. Second, the dropout claim is asserted without an experiment — they just say dropout 'degrades performance' and move on. Third, Table 2 compares against official reported baselines, not models trained in the same controlled setup, so those numbers are soft. Fourth, the 'trains faster' claim is confounded by the custom CUDA kernel, which fuses normalization and Fourier expansion; that's a systems optimization, not a property of the method.

The biggest gap, though, is the load-bearing assumption from Section 2: that frequency-ordered BPE token IDs carry structure the Fourier basis can exploit. The stress-test note puts it well — a token-permutation ablation would settle whether the ordering matters, and without it, the shared MLP plus residual connection could be doing all the work, with the Fourier basis essentially incidental. The paper does flag the near-collision problem in Section 4, but it defers the validation to future work, which leaves the central mechanism unsupported.

On citation pattern: the embedding-compression literature is cited, but they miss the Fourier feature line of work (sinusoidal positional encodings, random Fourier features). That's not a fatal flaw, but it weakens the novelty claim — this is a direct transfer of a known technique, not a new construction.

My verdict: conditional. The idea could be interesting at scale, and the proof-of-concept is worth taking seriously. But the paper needs variance estimates, controlled baselines, and that permutation ablation before the central claim is solid. Who is this for? Someone working on parameter-efficient architectures or efficient serving might find it a useful starting point. I'd send it to peer review — a workshop or short paper track would be appropriate — because it's reproducible and the question is worthwhile, but I'd expect substantial revision.

Referee Report

4 major / 7 minor

Summary. The paper proposes replacing the learned embedding lookup table in transformers with a deterministic Fourier expansion of normalized token IDs, followed by a shared lightweight MLP and a residual connection. The method is motivated by the hypothesis that frequency-ordered BPE token IDs carry statistically exploitable structure. The authors train standard transformers and their Fourier-embedding variant on SNLI and MNLI using a contrastive loss, evaluate zero-shot on STS-B, and report that the Fourier models achieve competitive performance with substantially fewer parameters and shorter training times. They also report fine-tuned PETE models in Table 2 with parameter counts as low as 58k, compared against official tiny-BERT baselines. The paper includes code and pre-trained weights.

Significance. If the core mechanism holds, the approach is relevant to memory-constrained and edge deployments, since embedding tables dominate parameter counts in large-vocabulary models. The paper has clear strengths: it ships reproducible code and weights, the zero-shot STS-B evaluation is a genuine out-of-distribution transfer after contrastive training on SNLI/MNLI, the Fourier basis is fixed and not tuned to the evaluation metric, and Table 1 provides a within-paper matched comparison under a shared training regime. The reported parameter reductions (e.g., 1.1m vs 8.9m parameters at comparable accuracy) are potentially valuable. However, the evidence as presented does not yet establish the central mechanism: the paper's own hypothesis about BPE ID ordering is untested, the reported gaps in Table 1 are within the range of run-to-run noise, and the universal approximation argument contains a mathematical gap. With additional controls and statistical rigor, the contribution could be solid; in its current form the claims outrun the evidence.

major comments (4)

[Section 3, Table 1] The central claim of competitiveness rests on differences of 1 to 3 Spearman points (e.g., 76.38 vs 77.34 for the 2-layer d=256 models, and 74.93 vs 77.01 for the 1-layer d=256 models). No error bars, confidence intervals, or repeated seeds are reported for any entry. In contrastive sentence-embedding training, seed-to-seed variance of this magnitude is typical, so the gaps may be within noise. I request repeated runs with at least three seeds and reporting of mean and standard deviation for the main Table 1 comparisons.
[Section 2 and Section 4] The paper's motivating hypothesis is that frequency-ordered BPE token IDs contain exploitable structure, and Section 2 explicitly states that the effectiveness of the fixed basis 'relies entirely' on this empirical hypothesis. However, no experiment isolates this factor. A token-permutation ablation—randomly permuting token IDs before normalization while keeping the Fourier features and MLP identical—would directly test whether the ordering matters. Without this control, the small gaps in Table 1 could be explained by the MLP and residual connection doing the actual work, with the specific Fourier ordering being incidental. Section 4 defers near-collision validation to future work, but the permutation test is a simple, load-bearing control that should be included now.
[Section 2, T_i(p) definition and universal approximation argument] The universal approximation argument invokes phi^{-1}(z), where phi(x)=T(p), but the Fourier map T is not injective on [-1,1]: at x=-1 and x=1, all sine terms vanish and all cosine terms coincide, so T(-1)=T(1) for every embedding dimension. Consequently phi^{-1} is undefined, and the final embedding E(p)=MLP(T(p))+T(p) provably cannot distinguish token IDs 0 and V-1. This is a concrete mathematical defect in the theoretical justification. The authors should either handle the endpoints separately (e.g., by excluding one endpoint or adding a small symmetry-breaking term) or weaken the universal approximation claim.
[Section 3, Table 2] The fine-tuned comparison in Table 2 uses official reported baselines (BERT-Tiny, BERT-Mini, TinyBERT, etc.) without any control for task-specific heads, training data, optimizer settings, or fine-tuning protocol. The statement in Section 5 that 'in configurations with equal overall parameters, PETE even outperforms the traditional transformer' is not supported by this uncontrolled comparison. A controlled fine-tuning experiment under identical settings is needed before such a claim can be made.

minor comments (7)

[Section 2] Clarify whether the index i in the Fourier basis definition is 0-based or 1-based, and state how odd d_model values are handled; the current definition is ambiguous for odd dimensions.
[Section 2 and Section 3] The claim that 'applying dropout to these normalized continuous mappings disrupts the smooth progression of token IDs and degrades performance' is asserted without any supporting experiment. Either add a dropout ablation or present this as a qualitative observation rather than an empirical result.
[Section 3] The text says the baseline and Fourier models differ 'only in the embedding layer,' but the baseline uses dropout 0.1 while the Fourier model omits dropout. This makes the Table 1 comparison confounded; the experimental setup should be stated precisely, or a dropout-matched baseline should be included.
[Section 3, Table 1] Training times are reported in minutes and hours without specifying the hardware configuration, whether times include data loading and evaluation, or how the custom CUDA kernel affects the comparison. Please provide these details.
[References] References [1] (Bertschinger et al.) and [19] (Williams and Beer) are not cited in the body text, and references [17] and [21] have incomplete author lists ('Wang and Others'). The bibliography should be tidied.
[Section 5] The claim that PETE outperforms traditional transformers when total parameters are equal is not demonstrated by Table 1 (which compares different parameter counts) or Table 2 (which is uncontrolled). Either add a controlled equal-parameter experiment or remove this sentence.
[Abstract] The abstract calls the embedding scheme 'parameter-free,' but the method includes a learnable MLP. Consider phrasing such as 'embedding-table-free' or 'deterministic base mapping' to avoid overstating the parameter reduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Fourier embedding is a fixed deterministic map, the MLP is trained by gradient descent, and the zero-shot STS-B evaluation is genuine out-of-distribution transfer.

full rationale

The paper's derivation chain is self-contained and not circular. The token representation is defined explicitly as E(p) = MLP(T(p)) + T(p), where T(p) is a fixed Fourier expansion of the normalized token ID and the MLP is a learned residual component. No fitted parameter is renamed as a prediction: the contrastive loss is trained on SNLI and MNLI, and STS-B scores are reported zero-shot and after fine-tuning, not computed from any fitted quantity in the method. The efficiency claim compares measured parameter counts and training times between architectures that differ only in the embedding layer, which is an experimental comparison rather than a definitional equivalence. The paper's central assumption that frequency-ordered BPE IDs contain exploitable statistical structure is an empirical hypothesis, explicitly acknowledged in Section 2 and revisited in Section 4 as a limitation; an untested premise is a correctness or scalability risk, not circularity. No load-bearing step invokes a self-citation or a uniqueness theorem, and the universal approximation remark is a standard mathematical statement rather than a derivation of the experimental result. Therefore no circular step can be identified from the paper's own equations or quoted text.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. It relies on the untested premise that token ID frequency ordering carries semantic structure, and on two standard mathematical tools (Fourier bases and MLP universal approximation). The hand-chosen hyperparameters are ordinary architecture choices, not constants fitted to the evaluation metric.

free parameters (4)

Embedding dimension d_model = 256 or 512
Chosen by hand as the width of the Fourier expansion and model; not fitted to the evaluation metric, but it directly sets the number of Fourier basis terms.
Number of transformer layers = 1 or 2
Chosen by hand to define the architecture scale in the experiments.
Feed-forward expansion factor = 4 (dim x 4) or 1/4 (dim / 4)
The intermediate dimension of the MLP blocks is a hand-chosen ratio that affects parameter counts in Table 2.
Dropout probability for Fourier models = 0
The authors state they observed dropout degrades performance, so they omit it for Fourier models, but no controlled dropout ablation is shown.

assumptions (4)

domain assumption BPE and WordPiece token IDs are assigned approximately in corpus frequency order, so the normalized ID x preserves useful statistical structure.
Section 2: the method 'relies entirely on the empirical hypothesis that the statistically-driven ordering of BPE token IDs contains exploitable structure that correlates, even weakly, with the desired final embeddings.' This is the core premise for using a Fourier basis.
ad hoc to paper The ideal token embedding function f: [-1,1] -> R is continuous enough for Fourier approximation and MLP residual approximation to apply.
The universal approximation argument in Section 2 assumes the target embedding varies smoothly with the normalized ID. There is no evidence that semantically meaningful token embeddings vary continuously with frequency-rank ID.
domain assumption Contrastive training on SNLI and MNLI transfers to sentence textual similarity on STS-B.
The experimental design in Section 3 trains on entailment pairs and evaluates on STS-B zero-shot; this is a standard transfer assumption but is not argued for.
standard math The MLP universal approximation theorem applies to the residual function H(z) = f(phi^{-1}(z)) - z.
Section 2 invokes the standard result that a multilayer perceptron can approximate continuous functions on a compact domain; the paper does not provide a proof, but this is a known theorem.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Parameter-Efficient Transformer Embeddings." pith.science (2026). https://pith.science/paper/OFRIDK3X

@misc{pith2026250502266,
  author       = {Pith},
  title        = {Pith review of: Parameter-Efficient Transformer Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OFRIDK3X}},
  note         = {Machine review of arXiv:2505.02266}
}

read the original abstract

Embedding layers in transformer-based NLP models typically account for the largest share of model parameters, scaling with vocabulary size but not yielding performance gains proportional to scale. We propose an alternative approach in which token embedding vectors are first generated deterministically, directly from the token IDs using a Fourier expansion of their normalized values, followed by a lightweight multilayer perceptron (MLP) that captures higher-order interactions. We train standard transformers and our architecture on natural language inference tasks (SNLI and MNLI), and evaluate zero-shot performance on sentence textual similarity (STS-B). Our results demonstrate that the proposed method achieves competitive performance using significantly fewer parameters, trains faster, and operates effectively without the need for dropout. This proof-of-concept study highlights the potential for scalable, memory-efficient language models and motivates further large-scale experimentation based on our findings.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages

[1]

Quantifying unique information

Nils Bertschinger, Jan Rauh, Eckehard Olbrich, Juliane Jost, and Nihat A y. Quantifying unique information. Entropy, 15(10):3508–3523, 2013

work page 2013
[2]

A large an- notated corpus for learning natural language inference

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Ch ristopher D Manning. A large an- notated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , 2015

work page 2015
[3]

From Wide to Deep: Dimension Lifting Network for Parameter-efficient Knowledge Graph Embedding

Borui Cai, Y ong Xiang, Longxiang Gao, Di Wu, He Zhang, Jio ng Jin, and Tom Luan. From wide to deep: Dimension lifting network for parameter-efﬁc ient knowledge graph embedding. arXiv preprint arXiv:2303.12816 , 2023

work page Pith review arXiv 2023
[4]

Bert: Pre-training of deep bidirectional transformers for language understan ding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina T outanova. Bert: Pre-training of deep bidirectional transformers for language understan ding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019

work page 2019
[5]

Albert: A lite bert for self-supervised learning o f language representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gi mpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning o f language representations. arXiv preprint arXiv:1909.11942, 2019

arXiv 1909
[6]

Root mean square layer normalization

Zhe Liu, Yixuan Lin, Zhiyuan Y ang, Peng Zhou, and Meizhu S un. Root mean square layer normalization. arXiv preprint arXiv:1910.07467 , 2020

arXiv 1910
[7]

Decoupled weight deca y regularization

Ilya Loshchilov and Frank Hutter. Decoupled weight deca y regularization. In International Conference on Learning Representations, 2019

work page 2019
[8]

Learning transferable visual models from natural la nguage supervision

Alec Radford, Jong Wook Kim, Christine Hallacy, Aditya R amesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mis hkin, and Jack Clark. Learning transferable visual models from natural la nguage supervision. https://arxiv.org/abs/2103.00020, 2021. arXiv preprint arXiv:2103.00020

arXiv 2021

Show all 21 references

[9]

Zero: Memory opti- mization towards training trillion parameter models

Samyam Rajbhandari, Jordan Rasley, Olatunji Ruwase, an d Y uxiong He. Zero: Memory opti- mization towards training trillion parameter models. arXiv preprint arXiv:1910.02054 , 2020

1910 arXiv
[10]

Neur al machine translation of rare words with subword units

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neur al machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association fo r Compu- tational Linguistics (ACL), 2016

2016
[11]

Glu variants improve transformer

Noam Shazeer. Glu variants improve transformer. In International Conference on Learning Representations, 2020

2020
[12]

Q-bert: Hessian-based ul tra low precision quantization of bert

Sheng Shen, Zhen Zhang, et al. Q-bert: Hessian-based ul tra low precision quantization of bert. In Proceedings of the Thirty-F ourth AAAI Conference on Artiﬁc ial Intelligence, 2020

2020
[13]

Compressing word emb eddings via deep compositional code learning

Ronghang Shu and Hideki Nakayama. Compressing word emb eddings via deep compositional code learning. arXiv preprint arXiv:1708.01079 , 2017

2017 arXiv
[14]

Roformer: Enhanced trans- former with rotary position embedding

Jun Su, Yiming Lu, Peng Huang, Cheng Xiong, and Jie Zhou. Roformer: Enhanced trans- former with rotary position embedding. In arXiv preprint arXiv:2104.09864 , 2021. 6

2021 arXiv
[15]

Hash embeddings for efﬁcient word representations

Dan Svenstrup and Jes Frellsen Hansen. Hash embeddings for efﬁcient word representations. In Advances in Neural Information Processing Systems , 2017

2017
[16]

Represe ntation learning with contrastive predictive coding

Aaron van den Oord, Y azhe Li, and Oriol Vinyals. Represe ntation learning with contrastive predictive coding. In Advances in Neural Information Processing Systems , 2018

2018
[17]

Structured embedding compression, 20 20

Wang and Others. Structured embedding compression, 20 20
[18]

A br oad-coverage challenge cor- pus for sentence understanding through inference

Adam Williams, Nandini Nangia, and Samuel R Bowman. A br oad-coverage challenge cor- pus for sentence understanding through inference. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , 2017

2017
[19]

Nonnegative decompo sition of multivariate information

Paul L Williams and Randall D Beer. Nonnegative decompo sition of multivariate information. arXiv preprint arXiv:1004.2515 , 2010

2010 arXiv
[20]

Mingxue Xu, Y ao Lei Xu, and Danilo P . Mandic. Tensorgpt: Efﬁcient compression of the embedding layer in llms based on the tensor-train decomp osition. arXiv preprint arXiv:2307.00526, 2023

2023 arXiv
[21]

Adaptively-masked twins-based layer f or efﬁcient embeddings, 2021

Y an and Others. Adaptively-masked twins-based layer f or efﬁcient embeddings, 2021. 7

2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Quantifying unique information

Nils Bertschinger, Jan Rauh, Eckehard Olbrich, Juliane Jost, and Nihat A y. Quantifying unique information. Entropy, 15(10):3508–3523, 2013

work page 2013

[2] [2]

A large an- notated corpus for learning natural language inference

Samuel R Bowman, Gabor Angeli, Christopher Potts, and Ch ristopher D Manning. A large an- notated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , 2015

work page 2015

[3] [3]

From Wide to Deep: Dimension Lifting Network for Parameter-efficient Knowledge Graph Embedding

Borui Cai, Y ong Xiang, Longxiang Gao, Di Wu, He Zhang, Jio ng Jin, and Tom Luan. From wide to deep: Dimension lifting network for parameter-efﬁc ient knowledge graph embedding. arXiv preprint arXiv:2303.12816 , 2023

work page Pith review arXiv 2023

[4] [4]

Bert: Pre-training of deep bidirectional transformers for language understan ding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina T outanova. Bert: Pre-training of deep bidirectional transformers for language understan ding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019

work page 2019

[5] [5]

Albert: A lite bert for self-supervised learning o f language representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gi mpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning o f language representations. arXiv preprint arXiv:1909.11942, 2019

arXiv 1909

[6] [6]

Root mean square layer normalization

Zhe Liu, Yixuan Lin, Zhiyuan Y ang, Peng Zhou, and Meizhu S un. Root mean square layer normalization. arXiv preprint arXiv:1910.07467 , 2020

arXiv 1910

[7] [7]

Decoupled weight deca y regularization

Ilya Loshchilov and Frank Hutter. Decoupled weight deca y regularization. In International Conference on Learning Representations, 2019

work page 2019

[8] [8]

Learning transferable visual models from natural la nguage supervision

Alec Radford, Jong Wook Kim, Christine Hallacy, Aditya R amesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mis hkin, and Jack Clark. Learning transferable visual models from natural la nguage supervision. https://arxiv.org/abs/2103.00020, 2021. arXiv preprint arXiv:2103.00020

arXiv 2021