Privacy-Preserving Transformers: SwiftKey's Differential Privacy Implementation

Abdelrahman Abouelenin; Amr Hendy; Mohamed Abdelrehim; Mohamed Afify; Raffy Fahim

REVIEW 3 major objections 5 minor 14 references

Privacy-Preserving Transformers: SwiftKey's Differential Privacy Implementation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Differentially private training lets a small transformer beat the production GRU in SwiftKey.

desk verdict The privacy guarantee is void: relaxed author sampling (Sec. 4.2) makes heavy users appear in nearly every batch, so the reported ε=14 is not a valid user-level bound, and the offline gains wash out in flight. read the letter →

arxiv 2505.05648 v1 pith:ML5WIV56 submitted 2025-05-08 cs.CL cs.CRcs.LG

Abdelrahman Abouelenin , Mohamed Abdelrehim , Raffy Fahim , Amr Hendy , Mohamed Afify This is my paper · ORCID

classification cs.CLcs.CRcs.LG

keywords differentialprivacyDP-SGDtransformerlanguagemodelnext-wordpredictionkeyboardtypingdataon-deviceinferenceONNXwordvocabulary

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a small transformer can be trained with user-level differential privacy for real keyboard language modeling and still outperform the production recurrent baseline. Using a scaled-down GPT-2-style decoder, the authors pretrain a seed model on general web and Twitter text, then DP-finetune it on anonymized typing data. They report consistent next-word-prediction and accuracy gains over the production 10k-vocabulary GRU on offline tests, with a modest increase in model size (about 20-25% at the same vocabulary) and inference time. The practical point is that differential privacy does not force a large quality penalty once the training recipe, sampling, and clipping are tuned.

What carries the argument

The load-bearing mechanism is a two-stage recipe: first build a seed model on general data (Common Crawl and Twitter), then fine-tune with DP-SGD on typing data from millions of users. Within DP fine-tuning, three components carry the results: relaxed author sampling (sampling sentences uniformly with at most one sentence per user per batch, to avoid a few heavy users dominating while preserving user-level DP), gradient clipping tuned to 0.01, and an $(\epsilon, \delta)$ budget with $\delta=10^{-8}$ and target $\epsilon$ around 14. The architecture is a downscaled GPT-2 decoder (4 layers, 4 attention heads, 512 hidden, 128 embedding, 20k word vocabulary, with a 64k-word unigram backoff for out-of-vocabulary words), quantized to bytes for on-device use.

What would settle it

Audit the privacy claim empirically: train two models with relaxed author sampling, one on data that includes a particular user's sentences and one without that user, then run a membership-inference test on the models; if the attacker's advantage exceeds what the claimed $(\epsilon, \delta)$ allows, the sampling scheme does not provide the stated privacy guarantee.

Watch

Extended reading notes

Core claim

The central claim is that a 4-layer, 4-head transformer decoder with hidden size 512, embedding size 128, and a 20k word vocabulary, trained by pretraining on public data and then DP-finetuning on typing data, achieves higher next-word-prediction and typing accuracy than the production GRU (10k vocabulary) in offline evaluations, and small edit-rate improvements for new users in flight. The model is quantized to roughly 6 MB and shipped through ONNX, adding about 20-25% memory over a same-vocabulary GRU and a gracefully larger mean inference time; a 12-layer version is much slower, so depth is the main cost. The same architecture also shows substantially lower pretraining loss than an equal-size GRU.

Load-bearing premise

The whole privacy guarantee rests on the claim that sampling sentences uniformly while allowing each user at most once per batch is still user-level differential privacy at the noise scale derived from standard DP-SGD; the paper gives no proof or citation for this modified sampling.

Editorial extensions

If this is right

A DP-finetuned 4-layer transformer can replace a recurrent LM in a production keyboard with better offline next-word-prediction and accuracy and only modest memory and latency increases.
Increasing vocabulary from 10k to 20k gives larger gains for the GRU than for the transformer, so the transformer's gains come partly from out-of-vocabulary reduction via the unigram backoff.
Longer contexts, as in the 50k BUS set, show larger gains than short snippets, suggesting the transformer's advantage grows with context length.
Deeper 12-layer transformers improve accuracy substantially but add too much latency for on-device use without caching, so caching and distillation are natural next steps.
Relative positional encoding provides the same accuracy as absolute encoding at SwiftKey context lengths while being robust to shifted position IDs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Pith inference: if relaxed author sampling does preserve user-level DP, then sentence-level uniform sampling with per-user caps may be applicable to other DP training pipelines to reduce the noise overhead of per-user sampling, though this deserves a formal privacy analysis.
Pith inference: the 20-25% memory increase at equal vocabulary suggests that embedding sharing or weight tying could shrink the transformer further, possibly to GRU size.
Pith inference: because the unigram backoff covers out-of-vocabulary words, a subword vocabulary might combine coverage with the transformer's longer-context strength and further close the DP/non-DP gap.
Pith inference: the flight results show the static LM's gains appear mainly for new users before the dynamic user model dominates, so future A/B tests should focus on cold-start settings.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is an honest engineering write-up from the SwiftKey team about DP-finetuning a small transformer for keyboard prediction, but the privacy claim does not survive contact with Section 4.2. I'll give them credit first.

The paper does several things well. It clearly describes the two-stage training (pretrain on public data, DP finetune on typing data), the architectural scaling (4L, 4A, 128-dim embeddings, 6MB quantized), and the ONNX deployment path. The offline comparisons across vocab sizes, data sizes, and positional encodings are useful engineering data. The FIRE relative positional encoding robustness test (Table 5) is a nice, clean result. The writing is candid about the flight results showing no aggregate improvement and about the DP vs non-DP gap.

Now the soft spots, and one of them is load-bearing.

The big one is 'relaxed author sampling' (Section 4.2). They sample sentences uniformly, with at most one per user per batch, and then use the standard DP-SGD accounting as if each user were sampled with probability q = L/U. That is not what the sampler does. A user with 67k sentences out of 628M total, with batch size 64k, appears in essentially every batch (inclusion probability ≈ 6.8, capped at 1). A user with 20 sentences appears with probability ≈ 0.002 per batch. The per-step amplification factor is the probability that a given user contributes to the step; for heavy users it is 1, not ~1e-4. Composing 28k steps with q≈1 for those users gives an ε far above 14—probably hundreds or more. The paper offers no proof, no citation, no modified accountant for this scheme. Since the entire point of DP here is to make users indistinguishable, this gap invalidates the central privacy claim. Entity anonymization does not fix it.

The empirical claims are also weaker than the abstract suggests. The offline tables show the DP transformer roughly matching the production GRU, not a consistent improvement; the only 'consistent gains' are in the 20k-vocab comparison against a 10k-vocab GRU, which is not an apples-to-apples baseline. The flight results show no aggregate edit-rate difference, and the age-group table is post-hoc slicing with no significance testing. Missing error bars and proprietary data make it hard to trust the small differences.

Bottom line: as an industrial case study, this is worth reading for its engineering choices and for the clear statement of what they tried. As a privacy paper, the core claim is unsupported. I would not cite it as evidence that DP training can be deployed at user-level with ε around 14. A referee should request either a proof for the sampler or, more realistically, a change to genuinely user-level sampling described in the paper. For a workshop on industrial NLP, it could pass with a heavy rewrite; for a main conference, the privacy flaw is disqualifying.

Referee Report

3 major / 5 minor

Summary. The paper reports an industrial implementation of a differentially private transformer language model for the SwiftKey keyboard. A GPT-2-style decoder is pre-trained on general data and then DP-fine-tuned on user typing data using Opacus/DP-SGD, quantized to about 6 MB, and integrated via ONNX. The authors compare 4-layer and 12-layer transformers with 10k and 20k vocabularies against production GRUs on next-word prediction and typing accuracy, report offline and flight results, and claim small and consistent gains over the production GRU with graceful memory and speed overhead.

Significance. If the privacy accounting were valid, this would be a valuable engineering case study showing that a small DP-fine-tuned transformer can be deployed on-device for keyboard language modeling. The paper has the strengths of a real deployment, concrete architecture choices, a comparison against a production baseline, and a clear separation of offline and flight measurements. However, the central privacy claim depends on a sampling scheme whose DP guarantee is not established, and the reported empirical gains are mixed and lack statistical support. As written, the contribution is a useful system description but not a validated demonstration of a privacy-preserving transformer with the claimed accuracy properties.

major comments (3)

[Section 4.2, 'Relaxed Author Sampling'] The sampling scheme described in this section invalidates the claimed user-level DP guarantee. DP-SGD accounting, as used in Section 3.4.2 with Opacus, assumes each user is included in a batch with probability at most q, typically via uniform or Poisson user sampling. Under the proposed 'relaxed author sampling', sentences are sampled uniformly and each user appears at most once per batch, but a user with many sentences has a much higher per-batch inclusion probability than a user with few sentences. With the reported numbers (a user with about 67k sentences, N=628M total sentences, and L=64k effective batch size), such a user appears in essentially every batch (inclusion probability close to 1), whereas a 20-sentence user appears with probability around 0.002. The amplification factor for heavy users is therefore effectively 1, not the small L/N value used in the accounting, and the reported epsilon of 14 is not a valid user-level privacy bound. The paper provides no proof, citation, or modified privacy accountant for this sampling scheme. Since this is the only mechanism described for fine-tuning on recent typing data, the central privacy claim is unsupported.
[Tables 2 and 3, Section 3.6.1] The abstract and summary claim 'small and consistent gains' against the production GRU, but the offline tables do not consistently support this. The main favorable comparison is between a 20k-vocabulary transformer and a 10k-vocabulary production GRU, so the vocabulary difference confounds the architecture comparison. In the same-vocabulary comparison shown in Table 2, the 4L4A transformer trained on 300M sentences is worse than the 20k-vocabulary GRU trained on 600M sentences on all reported metrics (e.g., 16.31 vs 17.32 NWP on Snippets and 27.44 vs 28.59 NWP on 50k BUS). No confidence intervals, standard errors, or significance tests are provided, so even the favorable differences such as 17.34 vs 17.32 are not shown to be reliable.
[Section 3.6.2, Figure 5 and Table 4] The flight results do not demonstrate a consistent gain. Aggregated edit rates in Figure 5 show essentially no difference between the transformer and the production GRU, and the age-group breakdown in Table 4 shows mixed results: some groups favor the transformer, one is identical, and the observed differences are tiny. This age-group slicing is post hoc and is presented without any pre-specified analysis or multiple-testing correction. The explanation that the dynamic user model 'starts to kick in' and overrides static model gains is speculative and not supported by the reported data. The flight evidence therefore cannot be used to substantiate the paper's central empirical claim.

minor comments (5)

[Section 3.4.2] There is a typo: 'Opcaus' should be 'Opacus'.
[Section 3.5 and Figure 3] There are typos: 'transofrmer' in the Figure 3 caption and 'transfomer' in the text; both should be 'transformer'.
[Section 2.2] The formula for the Gaussian mechanism noise scale is garbled: it should read roughly C * sqrt(2 log(1.25/delta)) / epsilon, and the scaling statement 'q epsilon sqrt(k)' is imprecise.
[Table 5] The table is hard to parse because the row labels 'no change' and 'shifted' are not defined in the text; please explain what 'abs rel' means and what the reported numbers represent.
[Section 3.1] The Snippets set is limited to a maximum of four commits for privacy, but the paper does not discuss whether this truncation biases the accuracy or NWP measurements; a brief note on this would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy, NWP, and speed comparisons are empirical and self-contained, with no fitted parameter or self-citation chain doing the derivational work.

full rationale

The paper's claims are empirical comparisons of DP-trained transformers against production GRU baselines. The transformer is a scaled GPT-2 trained with standard DP-SGD as implemented in the externally maintained Opacus library; the DP analysis is cited to Abadi et al. and implemented in Opacus, not derived from the present authors' prior results. Hyperparameters such as batch size, learning rate, and gradient clipping are tuned on held-out evaluation sets and reported as such, not fitted to the test sets whose metrics are claimed. Vocabulary-size choices are evaluated on an OOV test set and do not define the accuracy or NWP outcome by construction. The 'relaxed author sampling' of Section 4.2 is a potential privacy-correctness gap because the paper does not prove that its sentence-uniform-with-user-cap sampler satisfies the uniform-user-sampling assumption of the DP-SGD accountant; however, that is a correctness risk, not circularity, since the accuracy and speed claims do not reduce to the privacy accountant's inputs. No load-bearing self-citation or uniqueness theorem is invoked, and no known empirical result is merely renamed as a new contribution. The central finding—small consistent offline and early-age flight gains over the production GRU—is an externally benchmarked empirical result, so the derivation chain is self-contained.

Assumptions & free parameters 9 free parameters · 4 assumptions · 0 invented entities

The paper rests on a chain of empirical choices: hyperparameters tuned on small eval sets, a privacy budget chosen by policy, and an unproven sampling scheme. The most fragile assumptions are that the relaxed author sampling still satisfies user-level DP and that the test sets capture real typing behavior.

free parameters (9)

Vocabulary size = 20,000 words
Selected from OOV analysis (Table 1) as a trade-off between coverage and model size.
Embedding dimension = 128
Found to give nearly identical performance to 256 in initial experiments, saving memory.
Number of layers and heads = 4 layers, 4 heads
Chosen to meet on-device speed and memory constraints while capturing enough capacity.
Effective batch size = 64K sentences
Tuned in Section 4.1; larger batches lower loss but consume the privacy budget faster.
Learning rate = 1e-4
Tuned in Section 4.1 for stable DP fine-tuning.
Gradient clipping norm = 0.01
Tuned in Section 4.3; lower values distort gradients, higher values add more noise.
Target epsilon = 14
Chosen privacy budget for the final model.
Delta = 1e-8
Standard small DP delta; fixed in initial experiments.
Typing data weight in vocabulary construction = 5
Found best in initial experiments (Section 3.2).

assumptions (4)

domain assumption Anonymizing typing data by replacing entities with a placeholder removes personal information.
Section 3.1 asserts this without supporting evidence or analysis of residual leakage.
domain assumption The snippets test set and the 50k BUS set are representative of real user typing behavior.
Section 3.1 describes these sets; no analysis of representativeness or sampling bias is provided.
domain assumption The keypress simulation used to compute simulated accuracy on the 50k BUS set reflects real typing accuracy.
Section 3.1 (footnote 2) notes accuracy is simulated with a keypress model, but the model is not described or validated.
ad hoc to paper The relaxed author sampling scheme preserves user-level differential privacy with the noise scale from Abadi et al.
Section 4.2 modifies the standard sampling without a privacy proof or citation; the DP guarantee of the final model depends on this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Privacy-Preserving Transformers: SwiftKey's Differential Privacy Implementation." pith.science (2026). https://pith.science/paper/ML5WIV56

@misc{pith2026250505648,
  author       = {Pith},
  title        = {Pith review of: Privacy-Preserving Transformers: SwiftKey's Differential Privacy Implementation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ML5WIV56}},
  note         = {Machine review of arXiv:2505.05648}
}

read the original abstract

In this paper we train a transformer using differential privacy (DP) for language modeling in SwiftKey. We run multiple experiments to balance the trade-off between the model size, run-time speed and accuracy. We show that we get small and consistent gains in the next-word-prediction and accuracy with graceful increase in memory and speed compared to the production GRU. This is obtained by scaling down a GPT2 architecture to fit the required size and a two stage training process that builds a seed model on general data and DP finetunes it on typing data. The transformer is integrated using ONNX offering both flexibility and efficiency.

Figures

Figures reproduced from arXiv: 2505.05648 by the authors.

**Figure 1.** Pretraining evaluation loss of GRU and transformer model with same number of parameters. Eval set consist of 100k sentences taken from pretrained data down to satisfy on-device memory and speed requirements. We build a 4-layer model with 4 attention heads and a hidden dimension of 512. The input embeddings take a substantial part of the total model size. Therefore, we limited the embedding dimension to 128. In i… view at source ↗

**Figure 2.** eval loss dp vs non dp 3.4.2 Model Training Model training, either pretraining or DP finetuning, is done using the DP transformer library 6 . This library has a flexible architecture that support huggingface models and allows training transformers in DP and non-DP modes by integrating Opcaus (Yousefpour et al., 2021) with huggingface. As mentioned above, we first train a seed model on general data then DP finetun… view at source ↗

**Figure 4.** Different LM Sizes model) is a part of multiple search techniques contributing to its probability distribution for the next word prediction. In Fluency the ONNX model is loaded and used as part of inference. Performance tests are conducted to evaluate the speed of the transformer model and compare it to the production GRU for typing. A dataset composed of 630 typing samples is used. The sample is incrementally giv… view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: Flight Results for GRU 10k vocab and Transformer 20k vocab trained on 600M training-set [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: DP finetuning evaluation loss with different batch size and learning rate, batch size 2048 and 4096 are superior than smaller batch sizes. that we should always use the lowest gradient clipping value possible. However, reducing the clipping norm value degrades the o…

**Figure 8.** Figure 8: effect of Grad clipping abs rel no change 30.04 29.13 shifted 24.24 30.09 [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 7.** Figure 7: comparison between author sampling and relaxed author sampling result in more noise being added. Due to this trade off, we concluded that using a grad clipping norm of 0.01 is suitable for our case. 5 Positional Encoding Our transformer model is trained using learnable…

**Figure 9.** Figure 9: Hit @3 on different at different lengths [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 5 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. https://doi.org/10.1145/2976749.2978318 Deep learning with differential privacy . In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS’16. ACM

arXiv 2016
[4]

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

arXiv 2024
[5]

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. http://arxiv.org/abs/1802.08232 The secret sharer: Evaluating and testing unintended memorization in neural networks

arXiv 2019
[6]

Chen and Joshua Goodman

Stanley F. Chen and Joshua Goodman. 1996. https://doi.org/10.3115/981863.981904 An empirical study of smoothing techniques for language modeling . In 34th Annual Meeting of the Association for Computational Linguistics, pages 310--318, Santa Cruz, California, USA. Association for Computational Linguistics

work page arXiv 1996
[7]

Jack Cook. 2023. A look at Apple’s new Transformer-powered predictive text model . https://jackcook.com/2023/09/08/predictive-text.html. [Online; accessed 23-June-2024]

work page 2023
[8]

Cynthia Dwork. 2006. Differential privacy. In Automata, Languages and Programming, pages 1--12, Berlin, Heidelberg. Springer Berlin Heidelberg

work page 2006

Show all 14 references

[9]

Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. 2023. Functional interpolation for relative positions improves long context transformers. In The Twelfth International Co...

2023
[10]

Shiva Nosouhian, Fereshteh Nosouhian, and Abbas Kazemi Khoshouei. 2021. https://doi.org/10.20944/preprints202107.0252.v1 A review of recurrent neural network architecture for sequence learning: Comparison between lstm and gru . Preprints

2021
[11]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[12]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. http://arxiv.org/abs/1706.03762 Attention is all you need . CoRR, abs/1706.03762

2017 arXiv
[13]

Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. 2021. Opacus: U ser-friendly differential privacy library in PyTorch . arXiv prepri...

2021 arXiv
[14]

Jiawei Zheng, Hanghai Hong, Xiaoli Wang, Jingsong Su, Yonggui Liang, and Shikai Wu. 2024. http://arxiv.org/abs/2402.15061 Fine-tuning large language models for domain-specific machine translation

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang

Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. https://doi.org/10.1145/2976749.2978318 Deep learning with differential privacy . In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS’16. ACM

arXiv 2016

[4] [4]

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

arXiv 2024

[5] [5]

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. http://arxiv.org/abs/1802.08232 The secret sharer: Evaluating and testing unintended memorization in neural networks

arXiv 2019

[6] [6]

Chen and Joshua Goodman

Stanley F. Chen and Joshua Goodman. 1996. https://doi.org/10.3115/981863.981904 An empirical study of smoothing techniques for language modeling . In 34th Annual Meeting of the Association for Computational Linguistics, pages 310--318, Santa Cruz, California, USA. Association for Computational Linguistics

work page arXiv 1996

[7] [7]

Jack Cook. 2023. A look at Apple’s new Transformer-powered predictive text model . https://jackcook.com/2023/09/08/predictive-text.html. [Online; accessed 23-June-2024]

work page 2023

[8] [8]

Cynthia Dwork. 2006. Differential privacy. In Automata, Languages and Programming, pages 1--12, Berlin, Heidelberg. Springer Berlin Heidelberg

work page 2006