Subword Language Model for Query Auto-Completion

Gyuwan Kim

REVIEW 2 major objections 6 minor 12 references

Subword Language Model for Query Auto-Completion

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that subword language models, not just character-level ones, can power query auto-completion at up to 2.5 times the speed with negligible accuracy loss.

desk verdict Solid applied QAC paper with real speedups and a useful new metric; the unvalidated training surrogate is a legitimate but addressable concern. read the letter →

arxiv 1909.00599 v1 pith:NIVBTGOJ submitted 2019-09-02 cs.CL

Gyuwan Kim This is my paper · ORCID

classification cs.CL

keywords queryauto-completionsubwordlanguagemodelbytepairencodingregularizationretracealgorithmmeanrecoverablelengthbeamsearchneural

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that query auto-completion, usually done with character-level language models, can be done with subword language models at up to 2.5 times the speed while keeping completion quality nearly unchanged. Subwords pack several characters into one token, so an autoregressive decoder needs far fewer steps. To keep that speed-up from breaking generation, the paper introduces a retrace algorithm that steps back across token boundaries when a user's prefix ends mid-subword, and a reranking step that sums probabilities of duplicate token sequences from beam search. On the AOL query log, the best subword models stay within 0.02 points of the character baseline on MRR and PMRR, with a smaller average decode length. The paper also proposes mean recoverable length (MRL), a metric that counts how many characters ahead the model can complete correctly, avoiding the need to sample prefix lengths for evaluation.

What carries the argument

The central machinery is the subword vocabulary plus three supporting mechanisms. Subwords (from byte-pair encoding or subword regularization) let one decoding step emit several characters, shortening the sequence and cutting floating-point work. The retrace algorithm considers cases R0 through RL where the last token completing the prefix ends r characters before the prefix end; for each r it starts beam search from p1:|p|-r and requires the next token to match the retraced characters, so deterministic BPE cannot get stuck at a wrong boundary. Reranking by approximate marginalization treats the final beam as a sample of token sequences, groups them by the query they spell, sums each group's probabilities, and re-sorts; this approximates the intractable sum over all segmentations of a query. The MRL metric defines recoverable length as the number of characters before the first position at which the ground-truth query disappears from the candidate list, then averages it over the test set.

What would settle it

Take a sample of short queries and compute the exact query likelihood by marginalizing over all token segmentations with a small subword vocabulary; if the exact top-10 ranking differs from the retrace-plus-reranking beam search's top-10 in a substantial fraction of cases, the decoding approximations are not actually approximating the intended objective. Alternatively, train the same subword model with a segmentation-marginalized training objective and compare MRR and PMRR against the heuristic pseg training; a material gain for the exact model would show the heuristic training prior is the load-bearing weakness.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that treating a query as a sequence of subwords rather than characters is not only a speed optimization but a viable mechanism for accurate neural query auto-completion, provided the decoder handles the one problem character models never face: the user's typed prefix can end in the middle of a subword. For deterministic byte-pair encoding, this boundary mismatch is severe; the retrace algorithm, which restarts generation a few characters before the prefix end and forces the next token to cover the remaining prefix, lifts BPE's MRR from .242 to .430. For stochastic subword regularization, reranking by approximate marginalization over beam outputs adds a further small gain. The paper reports up to 2.5 times faster decoding on CPU and GPU with less than 0.02 MRR/PMRR drop, and shows the subword model generalizes better on unseen queries than the character baseline on PMRR. It also argues that the new metric MRL gives a length-consistent ordering between methods, unlike MRR and PMRR, which shift with the query-length distribution.

Load-bearing premise

The training objective assumes the pre-learned segmentation distribution pseg(t|q) is a reliable stand-in for the model's posterior p(t|q;θ); if those two disagree, the subword model is trained on a surrogate signal and the reported generation quality may not persist.

Editorial extensions

If this is right

Subword language models with retrace can replace character-level language models in production query auto-completion: up to 2.5 times faster, under 0.02 MRR/PMRR drop, and more accurate on unseen queries when using subword regularization.
Retrace is necessary for deterministic byte-pair encoding: without it BPE's MRR is .242, while with R2 it rises to .430, nearly matching the character model.
Approximate-marginalization reranking is orthogonal to retrace and gives a further small MRR gain, and its cost is negligible because it only sums and sorts the final beam.
MRL gives a single number that orders models consistently across query lengths, removing the need to sample prefix lengths for rank-based metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper leaves implicit is applying retrace to other prefix-constrained generative tasks, such as code completion or speech recognition, where segmentation boundaries may not align with the observed prefix; the R2 result suggests a retrace step of two characters may suffice for many subword vocabularies.
The paper notes that most-popular-completion dominates on seen queries while language models dominate on unseen ones, so an obvious combined system could fuse the two, but the paper does not test such a hybrid.
MRL may need normalization by query length for fair cross-corpus comparison; the paper explicitly leaves this open, and a recoverable-fraction variant would be a direct testable extension.
The heuristic choice of training with the pre-learned segmentation distribution is the weakest link, so one could test whether replacing it with a distillation objective from the character-level teacher closes the remaining gap.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a solid applied paper. The speedups are real, the evaluation is more thorough than most QAC papers, and the retrace algorithm and MRL metric are genuinely useful. The main soft spot is exactly what the stress-test flags: Section 3.2 swaps the true posterior for a pre-learned segmentation distribution and never validates that choice. That said, the empirical trade-off still holds up, and the paper is honest about the approximation.

What's new: first to apply subword LMs (BPE and subword regularization) to query auto-completion, plus a retrace algorithm that fixes the prefix-boundary mismatch (essential for BPE), reranking by approximate marginalization over duplicate beam outputs (small but real gain for SR), and a new metric, mean recoverable length, that removes prefix-length sampling. Good contributions.

What the paper does well: controlled comparison using the same LSTM size across Char/BPE/SR; time-based split of AOL; reports seen/unseen; ablates retrace depth and vocabulary size; code is public. The claim that accurate configs stay within ~0.02 MRR/PMRR at 1.4-2.4x speed is supported by Table 1. The 'up to 2.5x' in the abstract is technically true but comes from the BPE variant with the biggest accuracy drop; the configs that matter are the 1.4-2.4x ones. Minor packaging issue.

The soft spots, in order of importance. First, the training surrogate. The unbiased gradient estimator in Section 3.2 requires sampling from p(t|q;θ), but the paper uses pseg(t|q), calls it heuristic, and never ablates it. Section 7 admits it 'deteriorates the accuracy,' but we don't know how much of the reported gap comes from this. It's a correctness risk for the claimed speed/quality trade-off, because every subword model trains this way. Still, this is not fatal: the model demonstrably works on AOL, and the paper is upfront. A revision should compare against exact marginalization on a subset or at least measure how far pseg is from the model's posterior.

Second, no error bars. Differences of 0.01 in MRR/PMRR are being compared across single runs; reporting variance across seeds would strengthen the conclusions. Minor.

Third, MRL is a nice definition, but it ignores candidate order and hasn't been validated against user behavior; the paper acknowledges some of this. Fine for a first proposal.

The citation pattern is fine; related work on subword NMT and segmentation-aware LMs is covered. Overall, this is a serious, useful contribution. I'd bring it to reading group and would cite it if I worked on QAC or evaluation metrics. It deserves a real peer-review round; the training-surrogate issue is the thing to push on.

Referee Report

2 major / 6 minor

Summary. The paper proposes subword-level neural language models for query auto-completion (QAC). To handle the mismatch between prefix boundaries and subword token boundaries, it introduces a retrace algorithm that steps back a few characters from the end of the prefix and restricts decoding to match the retraced characters. For stochastic segmentation via subword regularization, it proposes reranking the beam outputs by approximate marginalization over duplicate segmentations. It also introduces a new evaluation metric, mean recoverable length (MRL). Experiments on the AOL query log show that the best subword configurations achieve 1.4–2.4x faster decoding than a character-level baseline while keeping MRR and PMRR within 0.02 points, with ablations isolating the effects of the retrace and reranking components.

Significance. If the empirical claims hold, this is a practical contribution: it demonstrates that subword language models are a viable alternative to character-level models for latency-sensitive QAC, with released code and reproducible ablations. The retrace and approximate-marginalization decoding techniques are well motivated, and their individual contributions are isolated in Table 1. The proposed MRL metric is a useful addition to the QAC evaluation toolbox, though its formal properties and reproducibility deserve further scrutiny. The main caveat is that the training objective rests on an unvalidated approximation, which weakens the strength of the central speed/quality trade-off claim.

major comments (2)

[Section 3.2, Eq. (1), Table 1] The training objective replaces the posterior p(t|q;θ) in the unbiased gradient estimator with the pre-learned pseg(t|q) 'heuristically,' but this substitution is never validated or ablated. Since Eq. (1) defines the query likelihood as a sum over segmentations, the subword LM's token-level probabilities are only meaningful for QAC if they approximate that query-level marginal; if pseg assigns mass to segmentations the LM cannot score well, the optimized objective is a biased surrogate and the observed generation quality in Table 1 could be an artifact of pseg being well-matched to AOL queries. The paper itself concedes in Section 7 that 'Approximation in training ... deteriorate the accuracy,' yet no experiment varies this choice, trains an exact-marginalization baseline, or estimates the divergence between pseg and p(t|q;θ). Because every subword model in Table 1 uses this surrogate, the central claim of 'similar quality' is not fully supported; I ask the authors to add a control or quantitative analysis of this approximation.
[Section 5.2, Table 1, Figure 2] The definition of recoverable length (RL) is informal and not sufficiently precise for reproduction. The statement that RL is 'the number of characters right before the first position where candidates do not have the query' leaves unspecified how the candidate list is generated at each prefix length, how ties are broken, and whether the metric is averaged over all prefixes or a single prefix per query. Without a precise algorithmic specification, the MRL results reported in Table 1 and Figure 2 cannot be independently verified. Please provide a formal definition, including the candidate-set construction and the stopping rule, and state how prefix lengths are enumerated.

minor comments (6)

[Section 4.1] The definition of the retrace case Rr is ambiguous; please formalize it with an equation, e.g., 'Rr is the set of token sequences whose last token overlapping with the prefix ends r characters before the end of the prefix.'
[Section 6.1] The paper truncates training queries to length 40, but does not state how test queries longer than 40 characters are handled during evaluation; this should be clarified because it can affect the reported metrics.
[Table 1] The column header 'Execution Speed (QPS)' uses two units; report this column simply as 'QPS' or 'Queries per second'.
[Figure 2] The left and right panels of Figure 2 are not explicitly labeled in the caption; please state explicitly that the left column varies query length and the right column varies prefix length.
[Section 5.2] The claim that MRL 'eliminates the need for prefix length sampling' is somewhat misleading, because computing MRL still requires enumerating prefix lengths; it eliminates the need for a sampling distribution but not for a search over prefix lengths.
[Author affiliation] There is a typo in the author affiliation: 'N AVER Corp.' should be 'NAVER Corp.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the subword QAC claims are supported by external benchmarks and standard metrics, and the acknowledged training approximation is a correctness risk, not a circular derivation.

full rationale

The paper derives query likelihood as a marginal over segmentations (Section 3) and then uses tractable approximations: the retrace algorithm (Section 4.1), beam-search reranking by approximate marginalization (Section 4.2), and the training-time substitution of pseg(t|q) for the posterior p(t|q;theta) (Section 3.2). None of these reduces a prediction to an input. The training substitution is explicitly labeled heuristic and its accuracy impact is acknowledged in Section 7 ('Approximation in training (Section 3.2) and decoding (Section 4) deteriorate the accuracy of subword language modeling'), which makes it a stated limitation rather than a concealed equivalence. The evaluation anchors to the external AOL query log and to the established MRR/PMRR baselines; no fitted constant is later reported as a prediction. The proposed MRL metric is a definition of a new evaluation statistic, not a model-derived quantity, and the paper also reports the standard metrics, so it cannot force the central claim. The retrace and reranking approximations are ablated against the character-level baseline in Table 1, providing independent empirical support. Self-citations are limited to infrastructure (NSML) and to standard external methods (SentencePiece, subword regularization), and none is load-bearing. Therefore the central speed/quality claim is self-contained and externally checkable.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard autoregressive language modeling plus three paper-specific choices: the segmentation-prior approximation in training, the beam-dominance assumption for reranking, and the finite retrace bound. These are disclosed by the author and are the main places where the empirical outcome could diverge from the stated method.

free parameters (5)

Subword regularization sampling exponent α = 0.2
Tuned on validation generation accuracy (Section 6.2); sets how peaked the segmentation sampling is during SR training.
Subword vocabulary size = 256
Chosen to balance performance and compute (Section 6.2); results vary with vocab size, especially BPE.
Beam size B = 30
Decoding and reranking beam width (Section 6.2); the marginalization reranking's quality depends on this.
Retrace step limit L = ∞ (R2 for BPE)
Maximum characters retraced from prefix end (Section 4.1); R2 is deemed sufficient from subword length distribution (Section 6.3).
LSTM capacity (hidden 600, embedding 100) = 600/100
Identical for all models 'for fairness' (Section 6.2); quality parity is conditional on this capacity.

assumptions (5)

standard math Autoregressive factorization p(t;θ) = ∏_i p(t_i|t_<i;θ) defines the language model over token sequences.
Used throughout Section 3 as the definition of the model family; no alternative is considered.
standard math The query probability is the sum over segmentations, p(q;θ) = ∑_{t∈S(q)} p(t;θ) (Eq. 2).
This is the latent-decomposition model from Chan et al. (2016) adopted as the formal basis for subword QAC.
ad hoc to paper pseg(t|q) approximates the posterior p(t|q;θ) for the training gradient.
Section 3.2: 'we heuristically use pseg(t|q) instead'; the author defers better approximations to future work. This is the main unvalidated modeling substitution.
domain assumption Beam-output probabilities dominate all non-beam probabilities (log p(t) ≫ log p(t′) for t in beam, t′ outside).
Section 4.2 states this assumption explicitly and uses it to justify approximate marginalization; no empirical check of the gap is reported.
domain assumption Retracing L characters covers all boundary mismatches of the target completions.
Section 4.1 limits retrace to L for computational reasons; Section 6.3 claims R2 ≈ R∞ based on subword length distribution, but no distribution data is shown.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Subword Language Model for Query Auto-Completion." pith.science (2026). https://pith.science/paper/NIVBTGOJ

@misc{pith2026190900599,
  author       = {Pith},
  title        = {Pith review of: Subword Language Model for Query Auto-Completion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NIVBTGOJ}},
  note         = {Machine review of arXiv:1909.00599}
}

read the original abstract

Current neural query auto-completion (QAC) systems rely on character-level language models, but they slow down when queries are long. We present how to utilize subword language models for the fast and accurate generation of query completion candidates. Representing queries with subwords shorten a decoding length significantly. To deal with issues coming from introducing subword language model, we develop a retrace algorithm and a reranking method by approximate marginalization. As a result, our model achieves up to 2.5 times faster while maintaining a similar quality of generated results compared to the character-level baseline. Also, we propose a new evaluation metric, mean recoverable length (MRL), measuring how many upcoming characters the model could complete correctly. It provides more explicit meaning and eliminates the need for prefix length sampling for existing rank-based metrics. Moreover, we performed a comprehensive analysis with ablation study to figure out the importance of each component.

Figures

Figures reproduced from arXiv: 1909.00599 by the authors.

**Figure 2.** Comparison of the character-level baseline model and our best models by changing query length and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages

[2]

arXiv preprint arXiv:1611.01462

Tying word vectors and word classiﬁers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462. Aaron Jaech and Mari Ostendorf. 2018. Personalized language model for query auto-completion. arXiv preprint arXiv:1804.09661. Kazuya Kawakami, Chris Dyer, and Phil Blunsom

arXiv 2018
[10]

NSML: A Machine Learning Platform That Enables You to Focus on Your Models

Blockwise parallel decoding for deep autore- gressive models. In Advances in Neural Information Processing Systems, pages 10106–10115. Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Donghyun Kwak, Jung-Woo Ha, et al. 2017. Nsml: A machine learning platform that enables you to focus on your model...

work page Pith review arXiv 2017
[12]

In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3674–3683

Sequence modeling via segmentations. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3674–3683. JMLR. org. Po-Wei Wang, J Zico Kolter, Vijai Mohan, and Inder- jit S Dhillon. 2018. Realtime query completion via deep language models. SIGIR eCom. Ronald J Williams. 1992. Simple statistical gradient- following algorith...

arXiv 2018
[152]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin

ACM. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998–6008. Chong Wang, Yining Wang, Po-Sen Huang, Abdel- rahman Mohamed, Dengyong Zhou, and Li Deng

work page 2017
[1058]

Gated Word-Character Recurrent Language Model

ACM. Yasumasa Miyamoto and Kyunghyun Cho. 2016. Gated word-character recurrent language model. arXiv preprint arXiv:1606.01700. Rodrigo Nogueira and Kyunghyun Cho. 2017. Task- oriented query reformulation with reinforcement learning. arXiv preprint arXiv:1704.04572. Dae Hoon Park and Rikio Chiba. 2017. A neural lan- guage model for query auto-completion. ...

work page Pith review arXiv 2016
[2006]

In InfoScale, volume 152, page 1

A picture of search. In InfoScale, volume 152, page 1. Oﬁr Press and Lior Wolf. 2016. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by back- propagating errors. nature, 323(6088):533. Stanislau Semeniuta, Aliaksei Severyn, ...

arXiv 2016
[2012]

preprint (http://www

Subword language modeling with neu- ral networks. preprint (http://www. ﬁt. vutbr. cz/imikolov/rnnlm/char. pdf), 8. Bhaskar Mitra, Milad Shokouhi, Filip Radlinski, and Katja Hofmann. 2014. On user interactions with query auto-completion. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval , pages 1055–

work page 2014
[2015]

arXiv preprint arXiv:1508.07909

Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. Leslie N Smith. 2018. A disciplined approach to neu- ral network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simons...

arXiv 2018

Show all 12 references

[2016]

arXiv preprint arXiv:1610.03035

Latent sequence decompositions. arXiv preprint arXiv:1610.03035. Junyoung Chung, Kyunghyun Cho, and Yoshua Ben- gio. 2016. A character-level decoder without ex- plicit segmentation for neural machine translation. arXiv preprint arXiv:1603.06147. Alexis Conneau, Holger Schwenk,...

2016 arXiv
[2017]

Transactions of the Association for Computational Linguistics , 5:365–378

Fully character-level neural machine trans- lation without explicit segmentation. Transactions of the Association for Computational Linguistics , 5:365–378. Jason Lee, Elman Mansimov, and Kyunghyun Cho
[2018]

arXiv preprint arXiv:1802.06901

Deterministic non-autoregressive neural se- quence modeling by iterative reﬁnement. arXiv preprint arXiv:1802.06901. Wang Ling, Edward Grefenstette, Karl Moritz Her- mann, Tom ´aˇs Ko ˇcisk`y, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016. Latent predic- tor networks for c...

2016 arXiv
[2019]

In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, pages 6429–6441

Learning to discover, ground and use words with segmental neural language models. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, pages 6429–6441. Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo J...

2018 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [2]

arXiv preprint arXiv:1611.01462

Tying word vectors and word classiﬁers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462. Aaron Jaech and Mari Ostendorf. 2018. Personalized language model for query auto-completion. arXiv preprint arXiv:1804.09661. Kazuya Kawakami, Chris Dyer, and Phil Blunsom

arXiv 2018

[2] [10]

NSML: A Machine Learning Platform That Enables You to Focus on Your Models

Blockwise parallel decoding for deep autore- gressive models. In Advances in Neural Information Processing Systems, pages 10106–10115. Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngkwan Kim, Gayoung Lee, Donghyun Kwak, Jung-Woo Ha, et al. 2017. Nsml: A machine learning platform that enables you to focus on your model...

work page Pith review arXiv 2017

[3] [12]

In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3674–3683

Sequence modeling via segmentations. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3674–3683. JMLR. org. Po-Wei Wang, J Zico Kolter, Vijai Mohan, and Inder- jit S Dhillon. 2018. Realtime query completion via deep language models. SIGIR eCom. Ronald J Williams. 1992. Simple statistical gradient- following algorith...

arXiv 2018

[4] [152]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin

ACM. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998–6008. Chong Wang, Yining Wang, Po-Sen Huang, Abdel- rahman Mohamed, Dengyong Zhou, and Li Deng

work page 2017

[5] [1058]

Gated Word-Character Recurrent Language Model

ACM. Yasumasa Miyamoto and Kyunghyun Cho. 2016. Gated word-character recurrent language model. arXiv preprint arXiv:1606.01700. Rodrigo Nogueira and Kyunghyun Cho. 2017. Task- oriented query reformulation with reinforcement learning. arXiv preprint arXiv:1704.04572. Dae Hoon Park and Rikio Chiba. 2017. A neural lan- guage model for query auto-completion. ...

work page Pith review arXiv 2016

[6] [2006]

In InfoScale, volume 152, page 1

A picture of search. In InfoScale, volume 152, page 1. Oﬁr Press and Lior Wolf. 2016. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by back- propagating errors. nature, 323(6088):533. Stanislau Semeniuta, Aliaksei Severyn, ...

arXiv 2016

[7] [2012]

preprint (http://www

Subword language modeling with neu- ral networks. preprint (http://www. ﬁt. vutbr. cz/imikolov/rnnlm/char. pdf), 8. Bhaskar Mitra, Milad Shokouhi, Filip Radlinski, and Katja Hofmann. 2014. On user interactions with query auto-completion. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval , pages 1055–

work page 2014

[8] [2015]

arXiv preprint arXiv:1508.07909

Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. Leslie N Smith. 2018. A disciplined approach to neu- ral network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simons...

arXiv 2018