Foundations of Large Language Models

Jingbo Zhu; Tong Xiao

Foundations of Large Language Models

T0 review · 0 major / 9 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A five-stage pipeline explains large language models from pretraining to inference.

desk verdict A competent, clearly written textbook excerpt with no new research content; fine as teaching material, not a research preprint. read the letter →

arxiv 2501.09223 v2 pith:LA755XIY submitted 2025-01-16 cs.CL cs.AIcs.LG

Tong Xiao , Jingbo Zhu This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords largelanguagemodelspre-trainingself-supervisedlearninggenerativepromptingchain-of-thoughtalignmentinference

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This book argues that large language models are best understood as a five-stage pipeline: pre-training, generative modeling, prompting, alignment, and inference. It aims to give students and practitioners a self-contained introduction to these foundations. If the book is correct, a working knowledge of an LLM requires understanding how these stages connect, and the core mechanism is self-supervised token prediction at scale.

What carries the argument

The load-bearing machinery is the pre-train-then-adapt pipeline: a Transformer, usually decoder-only, is trained with a self-supervised language-modeling objective on large unlabeled text and then adapted by fine-tuning, prompting, alignment, and inference-time decoding. The concept that carries the argument is the next-token maximum-likelihood objective, which turns unlabeled text into supervision and makes knowledge acquisition a scaling problem. The book maps its five chapters onto this pipeline.

What would settle it

The central claim would be falsified if a reader following the book's BERT recipe (15% masking, the combined MLM and NSP loss, then fine-tuning on labeled data) could not reproduce the described classification behavior, or if the quoted scaling-law constants in Equation (2.39) differed from the cited source.

Watch

Extended reading notes

Core claim

On the book's own terms, the central discovery is that self-supervised token prediction on large unlabeled text produces a general-purpose language model that can then be steered without retraining from scratch. The book develops this by walking through decoder-only Transformers trained with next-token maximum likelihood; masked and denoising objectives for encoders and encoder-decoders; scaling laws linking loss to model size, data, and compute; prompting and chain-of-thought methods; alignment through instruction tuning and human-feedback reinforcement learning; and inference-time techniques such as decoding algorithms and cache management. The presentation is a coherent pipeline in which pre-training supplies world knowledge and the later stages give task-directed behavior.

Load-bearing premise

The book's value rests on the assumption that the five selected topics really are the foundational core of LLM practice and that the cited methods and numbers are represented faithfully.

Editorial extensions

If this is right

If the pipeline is the real structure of the field, failures in an LLM should be diagnosed by locating which stage, pre-training, generation, prompting, alignment, or inference, went wrong.
Scaling laws imply that loss will keep improving with model size, data, and compute along a predictable curve until irreducible error dominates, so compute and data budgets can be planned ahead of training.
The book's account of prompting implies that many tasks can be done by a single model without extra training, using demonstrations and chain-of-thought reasoning.
Alignment from human feedback is presented as necessary after pre-training and instruction tuning, so a deployment-ready LLM should be expected to undergo all three.
Long-context capability is not free; it requires modifications such as sparse or linear attention, cache management, and position extrapolation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the five-part structure gives both a curriculum and a debugging checklist; in an application, first identify which stage is failing.
Editorial inference: because the book notes that lower pre-training loss does not always mean better downstream performance, an obvious test is to see which downstream abilities track pre-training loss versus alignment or prompting effort.
Editorial inference: if pre-training is the real source of world knowledge, then adding reasoning-heavy data to the pre-training objective should improve later reasoning; the book does not test this, but its framework invites the experiment.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a textbook excerpt, not a research paper. The preface says it is five chapters selected from the authors' own NLPBook, and that is exactly what it reads like. No new science, method, or measurement; the value is pedagogical.

What it does well: the exposition is clear and well-structured. The math is standard and mostly correct. The examples — BERT, T5, RLHF, chain-of-thought — are concrete and useful. The comparison table of pre-training tasks (Table 1.1) is genuinely good teaching material, and the book is self-contained enough for a student with basic ML background. The underlying NLPBook is open, which is a plus.

I checked the specific concerns in the report. The GeLU definition in Section 2.2.2.2 is the standard Hendrycks-Gimpel form. The Chinchilla constants in Section 2.2.4 match Hoffmann et al. The summation-bound typo in Eq. (1.7) is real but trivial: the prose and Eq. (1.6) immediately give the correct maximum-likelihood objective. So the soundness worries mostly evaporate on inspection. The legitimate caveat is that fitted scaling laws are presented as exact equations without uncertainty or a note that they are empirical fits. For a textbook that's a minor flaw, but students will copy those constants.

Soft spots: zero novelty, by design. The book explicitly disclaims comprehensiveness, so the five-chapter selection is an announced editorial choice. There is minor self-citation to NLPBook, which is fine given provenance. A few presentation glitches (some awkward notation in the sequence-parallelism section), but nothing load-bearing.

Who is this for: instructors, students, and practitioners who want a compact introduction. Not for researchers seeking new results. As a submission to a research venue, it should be desk rejected. If the venue is an educational resource or textbook series, a light accuracy review would be worthwhile. My recommendation: don't spend referee time on it as a research contribution, but do use it as teaching material.

Referee Report

0 major / 9 minor

Summary. This manuscript is a five-chapter excerpt from the authors' larger NLP textbook, drawn from the publicly available NLPBook. It aims to outline the foundational concepts of large language models, covering pre-training (with BERT as a running example), generative decoder-only LLMs and training at scale, prompting, alignment, and inference. The exposition is example-driven and includes standard mathematical formalizations: chain-rule factorization of sequence probability, maximum-likelihood objectives for causal and masked language modeling, ranking losses for reward models, and equations for attention, parallelism, and long-sequence modeling. The preface and chapter summaries explicitly disclaim comprehensive coverage and position the book as an introduction for students and practitioners.

Significance. The book makes no claim to new research results, so its significance is pedagogical. Its strengths are clarity, consistent notation, and breadth: worked prompting examples, comparison tables of pre-training objectives and LLM configurations, and accessible treatments of RLHF and inference. Spot-checks of the technical core, including the MLE objectives, BERT-style losses, RLHF reward-model losses, and sparse and linear attention, found no substantive misstatements. The flagged GeLU formula is standard, and the Chinchilla constants match the cited source; the issues are presentation-level, not load-bearing. No machine-checked proofs or reproducible artifacts are claimed, but none are needed for this expository contribution.

minor comments (9)

[Sec. 1.2.1, Eq. (1.7)] The summation bound in the re-expression of the maximum-likelihood objective is typographically wrong: the inner sum should run from i=0 to m-1, not to i-1. Equations (1.5) and (1.6) state the correct objective, so this is a local typo, but it should be corrected in a revised version.
[Sec. 2.1.1, Eq. (2.6)] The causal-mask condition is reversed. As written, Mask(i,k)=0 when i<=k, which would let a query at position i attend to future keys; the intended mask allows k<=i, as the lower-triangular pattern in Eq. (2.46) and Figure 2.1 makes clear. Please reverse the inequality text.
[Sec. 2.2.2.2, Eq. (2.28)] In the GLU definition, the second multiplicative term is missing its input: sigma_glu(h)=sigma(hW1+b1) * (W2+b2) should read sigma(hW1+b1) * (hW2+b2). The same omission occurs in Eqs. (2.29) and (2.30).
[Sec. 2.2.2.2, GeLU paragraph] The sentence 'Here h is a d-dimensional vector whose entries are drawn from the standard normal distribution' is not the intended definition of GeLU. The formula sigma_gelu(h)=hPhi(h) is the standard Hendrycks-Gimpel form with Phi the standard-normal CDF applied entrywise; please rephrase so readers are not misled into thinking GeLU requires sampling.
[Sec. 2.2.4, Eqs. (2.36)-(2.39)] The fitted scaling laws are presented with exact constants but without confidence intervals, units, or a statement that the constants are point estimates from particular runs. The surrounding text already notes that scaling laws are empirical and not universal, so adding one sentence on uncertainty and on the units of N and D would remove the main risk of students treating these equations as exact.
[Sec. 2.1.3, Eq. (2.17)] The concatenation order in Eq. (2.17) appears reversed: the line 'sample = [y_sample, x_sample]' should be '[x_sample, y_sample]', since Eq. (2.18) defines the loss as -log Pr(y_sample | x_sample).
[Table 2.3] The row labeled 'LLaMA2-65B' is inconsistent with Table 2.2, which lists LLaMA2 at 70B; this should be either 'LLaMA-65B' (Touvron et al. 2023a) or 'LLaMA2-70B' with the correct token count, and the citation should be reconciled accordingly.
[Sec. 2.1.4] The phrase 'robust, secure, and subjective' appears to contain a typo; likely 'safe' or 'reliable' was intended. Also, in Section 1.4, the citation 'Wang et al., 2023a;e' contains a stray 'e'.
[Sec. 2.3.2] The text states that the first strand of long-sequence research 'can be found in general discussions of efficient Transformer architectures [Tay et al., 2020; Xiao and Zhu, 2023]'. Because Xiao and Zhu [2023] is the authors' own book, this is a self-reference to an external resource; a one-paragraph summary of the main efficient-attention families would make the excerpt more self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the book is an expository survey whose technical claims are attributed to external literature, and its self-citations are non-load-bearing provenance.

full rationale

The central assertion is pedagogical: the Preface says the book 'aims to outline the basic concepts of large language models and introduce the related techniques.' There is no derivation chain whose conclusion is assumed in its premise. Technical results are quoted from external primary sources: the scaling laws in Section 2.2.4 are explicitly attributed to Hestness et al. [2017], Kaplan et al. [2020], Rosenfeld et al. [2020], and Hoffmann et al. [2022], with fitted constants presented as empirical fits rather than as predictions derived by this book. The GeLU definition in Eq. (2.27) is the standard Hendrycks-Gimpel form, and the masked, permuted, and denoising objectives are standard maximum-likelihood constructions, not self-referential devices. The self-citations to Xiao and Zhu [2023] in Sections 1.3.3 and 2.3, and the pointers to the authors' NLPBook repository, are given as further reading and background provenance; they do not carry the book's central claim, and removing them would not change any technical statement. Since no fitted input is renamed as a prediction, no uniqueness theorem is imported from the authors, and no ansatz is smuggled in via citation, the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The book introduces no new entities. Its pedagogical claim rests on standard mathematics (probability, attention) and on the accuracy of external fitted constants and contested research claims (scaling laws, emergent abilities). The fitted constants are imported from the literature without error bars, so they are listed as free parameters in the ledger.

free parameters (2)

Chinchilla scaling law constants = a=406.4, b=410.7, epsilon=1.69, exponents 0.34 and 0.28
Quoted in Section 2.2.4 as an exact law; these values are fitted to empirical loss curves in Hoffmann et al. (2022), not derived in this book.
Kaplan scaling law constants = N_0=8.8e13, alpha=0.076; D_0=5.4e13, beta=0.095
Quoted in Section 2.2.4 as L(N)=(N/8.8e13)^-0.076 and L(D)=(D/5.4e13)^-0.095, empirical fits from Kaplan et al. (2020).

assumptions (4)

standard math Language modeling via next-token prediction is equivalent to maximum likelihood estimation (Eq. 1.7).
The book derives this from the chain rule of probability; this is a standard mathematical background result.
domain assumption The QKV self-attention formulation in Section 2.1.1, including the causal mask, faithfully represents decoder-only Transformers used in GPT, LLaMA, and others.
The book describes these architectures at a high level and does not verify against implementations.
domain assumption Quoted scaling-law constants (e.g., Chinchilla 406.4/N^0.34 + 410.7/D^0.28 + 1.69 and Kaplan L(N)=(N/8.8e13)^-0.076) are accurate.
Presented in Section 2.2.4 as established laws without derivation, uncertainty, or reproduction.
domain assumption Emergent abilities in LLMs are established facts following Wei et al. (2022b).
Stated in Section 2.2.4 without acknowledging the subsequent debate about metric artifacts and alternative explanations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Foundations of Large Language Models." pith.science (2026). https://pith.science/paper/LA755XIY

@misc{pith2026250109223,
  author       = {Pith},
  title        = {Pith review of: Foundations of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LA755XIY}},
  note         = {Machine review of arXiv:2501.09223}
}

read the original abstract

This is a book about large language models. As indicated by the title, it primarily focuses on foundational concepts rather than comprehensive coverage of all cutting-edge technologies. The book is structured into five main chapters, each exploring a key area: pre-training, generative models, prompting, alignment, and inference. It is intended for college students, professionals, and practitioners in natural language processing and related fields, and can serve as a reference for anyone interested in large language models.

Figures

Figures reproduced from arXiv: 2501.09223 by the authors.

**Figure 1.1.** Illustration of unsupervised, supervised, and self-supervised pre-training. In unsupervised pre-training, the pre-training is performed on large-scale unlabeled data. It can be viewed as a preliminary step to have a good starting point for the subsequent optimization process, though considerable effort is still required to further train the model with labeled data after pre-training. In supervised pre-training, the … view at source ↗

**Figure 1.2.** Pre-training a Transformer encoder (left) and then applying the pre-trained encoder (right). In the pre-training phase, the encoder, together with a Softmax layer, is trained via self-supervision. In the application phase, the Softmax layer is removed, and the pre-trained encoder is combined with a prediction network to address specific problems. In general, for better adaptation to these tasks, the system is fine-t… view at source ↗

**Figure 1.3.** Comparison of self-attention masking results of causal language modeling, masked language modeling and permuted language modeling. The gray cell denotes the token at position j does not attend to the token at position i. The blue cell (i, j) denotes that the token at position j attends to the token at position i. emask represents the embedding of the symbol [MASK], which is a combination of the token embedding and t… view at source ↗

Figures from the paper (52 more)

**Figure 1.4.** Figure 1.4: Training an encoder-decoder model using BERT-style and denoising autoencoding methods. In both methods, the input to the encoder is a corrupted token sequence where some tokens are masked and replaced with [MASK] (or [M] for short). The decoder predicts these masked …

**Figure 1.5.** Figure 1.5: A running example of BERT-style masked language modeling. First, 15% of the tokens are randomly selected. These selected tokens are then processed in one of three ways: replaced with a [MASK] token (80% of the time), replaced with a random token (10% of the time), or…

**Figure 1.6.** Figure 1.6: The model architecture of BERT (Transformer encoder). The input tokens are first represented as embeddings, each of which is the sum of the corresponding token embedding, positional embedding and segment embedding. Then, the embedding sequence is processed by a stack…

**Figure 1.7.** Figure 1.7: An illustration of translation language modeling. For ease of understanding, we present a simple example where all the selected tokens are masked. The model is trained to predict these masked tokens. As the sequence contains tokens in two languages, predicting a toke…

**Figure 2.1.** Figure 2.1: The Transformer-decoder architecture for language modeling. The central components are L stacked Transformer blocks, each comprising a self-attention sub-layer and an FFN sub-layer. To prevent the model from accessing the right-context, a masking variable is incorpo…

**Figure 2.2.** Figure 2.2: An overview of RLHF. There are 4 key steps involved: a) training an initial LLM (i.e., policy) using pretraining and supervised fine-tuning; b) collecting human preference data by ranking the outputs of the LLM; c) training a reward model using the ranking results; …

**Figure 2.3.** Figure 2.3: A scaling law of test error against a variable of interest (e.g., training dataset size) [Hestness et al., 2017]. The curve of the scaling law can be divided into three phases. At the beginning, the number of test errors decreases slowly when more training data is us…

**Figure 2.4.** Figure 2.4: Test loss against model size (N) and training dataset size (D) (data points are plotted for illustrative purposes). We plot test loss as a function of N, which is defined as L(N) = N 8.8×1013 −0.076, and a function of D, which is defined as L(D) = D 5.4×1013 −0.095…

**Figure 2.5.** Figure 2.5: Illustrations of self-attention, sparse attention, linear attention and recurrent models. Blue boxes = cached states for producing the output at position i. f(·) = a recurrent cell. tokens are kept, and the cost of caching grows as the inference proceeds. Above, we h…

**Figure 2.6.** Figure 2.6: Illustrations of fixed-size KV caches in LLMs. Blue boxes represent the keys and values generated during LLM inference, green boxes represent the keys and values stored or encoded in the primary memory, and orange boxes represent the keys and values stored or encoded…

**Figure 2.7.** Figure 2.7: Illustrations of external memories (or datastores) for language modeling [PITH_FULL_IMAGE:figures/full_fig_p086_2_7.png]

**Figure 2.8.** Figure 2.8: Illustration of QKV attention based on different multi-head and sharing mechanisms. (a) = single-head attention, and (b-e) = attention with multiple heads. directly accesses the KV cache of a lower-level layer. 2.3.5 Position Extrapolation and Interpolation Since Tra…

**Figure 2.9.** Figure 2.9: Illustrations of different positional embedding methods for a range of positions. Blue points represent the positions that have been observed during training, and red points represent the positions that are newly observed at test time. In sub-figure (a), the encoding…

**Figure 2.10.** Figure 2.10: Illustration of distributing query-key offsets into buckets in the T5 model (nb = 32 and distmax = 1024). Boxes represent buckets. In the first half of the buckets, we use a fixed bucket size. In the second half of the buckets, we increase the bucket size logarithmi…

**Figure 2.11.** Figure 2.11: Query-key products with biases (above = the T5 bias and below = the ALiBi bias). The color scale of the biases ranges from light blue denoting small absolute values to deep blue denoting large absolute values. attention weights α(i, j) = Softmax( qik T j + β · (j − …

**Figure 2.12.** Figure 2.12: Illustrations of vector rotations in a plane. Sub-figures (a) and (b) show rotations of a vector in a single step and multiple steps, respectively. Sub-figure (c) shows the embeddings of tokens cat and sleeping in two different sentences. We show these sentences wit…

**Figure 3.1.** Figure 3.1: CoT in four different reasoning tasks, including CSQA, StrategyQA, Dyck languages, and Last Letter Concatenation. The CoT parts are highlighted in green. the problem6 . There are generally no standard or clear paths to a solution for a complex problem. However, it …

**Figure 3.2.** Figure 3.2: Ensembling methods for LLMs. In standard model ensembling (a), multiple LLMs varying in architectures or parameters are used. Each LLM receives the same prompt and produces a prediction. These predictions are combined to generate the final prediction. In prompt ensem…

**Figure 3.3.** Figure 3.3: Illustration of hard and soft prompts. Here the hard prompt is the instruction we input to the LLM for performing the task. The LLM encodes this instruction as usual, and the intermediate representations corresponding to the instruction can be viewed as some sort of …

**Figure 3.4.** Figure 3.4: Illustration of context distillation [Snell et al., 2022]. The teacher model is a standard LLM, which takes both the context and the user input as model input and produces a prediction as model output. Then, we simplify the context (e.g., simplifying the instruction …

**Figure 3.5.** Figure 3.5: Illustration of prefix fine-tuning for a translation task (Look out! → 小心!). For each layer, we add two prefixes p l 0 and p l 1 at the beginning. The LLM is trained to minimize the loss on the predictions given the input. During this process, only the prefixes are o…

**Figure 3.6.** Figure 3.6: Illustration of prompt tuning for a translation task (Look out! → 小心!). Instead of using fixed textual prompts, soft prompts are learnable embeddings that are added at the beginning of the embedding sequence. During fine-tuning, only these prompt embeddings are optim…

**Figure 3.7.** Figure 3.7: Illustrations of using soft prompts in LLMs. Here tunable soft prompts are shown in blue, and components whose parameters are fixed during fine-tuning are shown in gray. In sub-figure (a), soft prompts are prefixes appended to each layer of the LLM. In sub-figure (b)…

**Figure 3.8.** Figure 3.8: Illustration of compressing a context segment into soft prompts (κ = 2 and mi = 4). The input to the LLM includes the soft prompts from the previous step (σ <i 1 and σ <i 2 ), the tokens of the segment (z1, z2, z3, and z4), and the summary tokens (⟨g1⟩ and ⟨g2⟩). Giv…

**Figure 4.1.** Figure 4.1: Schematic illustration of the pre-train-then-align method for developing LLMs. In the pre-training stage, we train an LLM on vast amounts of data using next token prediction. Then, in the alignment stage, we align the LLM to user instructions, intents, and preference…

**Figure 4.2.** Figure 4.2: Illustration of supervised fine-tuning for LLMs. We concatenate the input and the output into a single sequence. During the forward pass, we run the LLM as usual. During the backward pass, we compute the loss only for the output part and simply set the loss for the …

**Figure 4.3.** Figure 4.3: Illustration of supervised fine-tuning for conversational models. Here the LLM acts as a chatbot to respond to each request based on the conversational history. The conversation progresses by alternating between the user and the chatbot. In SFT, we treat the entire c…

**Figure 4.4.** Figure 4.4: Illustration of self-instruct [Wang et al., 2023b]. This method maintains a pool of instructions and corresponding input-output samples. Initially, the pool contains a number of hand-crafted instructions and samples. Each time, we draw a few instructions from the po…

**Figure 4.5.** Figure 4.5: Illustrations of using small models to improve large models in LLMs. One approach involves using smaller models for the fine-tuning or pre-training of larger models. This includes generating synthetic data (a), incorporating auxiliary loss (b), and selecting appropri…

**Figure 4.6.** Figure 4.6: Supervised fine-tuning vs. reinforcement learning from human feedback. In supervised fine-tuning, we optimize the LLM by maximizing the probability of the prediction given the input. In reinforcement learning from human feedback, we first train a reward model on huma…

**Figure 4.7.** Figure 4.7: LLM as policy in reinforcement learning. At each step t, the LLM predicts a token yt given the model input x and the previously-generated tokens y<t. This process can be framed as a reinforcement learning problem, where yt serves as the action, (x, y<t) as the state,…

**Figure 4.8.** Figure 4.8: Architecture of the reward model based on Transformer. The main component of this model is still an LLM. We use the Transformer decoder as the sequence representation model. We extract the representation of the last position of the decoder as the representation of th…

**Figure 4.9.** Figure 4.9: Illustration of RLHF. The first step is to collect human preference data and train the reward model using this data. Once the reward model is optimized, along with the reference model, we proceed to train both the policy and the value function. At each prediction ste…

**Figure 4.10.** Figure 4.10: Standard RLHF (PPO) vs. DPO. In RLHF, the human preference data is used to train a reward model, which is then employed in training the policy as well as the value function. In DPO, the use of human preference data is more direct, and the policy is trained on this d…

**Figure 4.11.** Figure 4.11: Two LLM responses to a math problem. In response 1, both the final result and all the reasoning steps are correct. In response 2, the final result is correct, but there are mistakes in the reasoning process (highlighted in red). For outcome-based approaches, both re…

**Figure 5.1.** Figure 5.1: The decoder-only architecture for LLMs. The decoder consists of an embedding layer and a stack of Transformer layers. In each Transformer layer, the input passes through a linear mapping, a self-attention network, and an FFN. The output of the decoder is a sequence …

**Figure 5.2.** Figure 5.2: Illustration of the KV cache. We update the KV cache at a position, perform the attention operation, and then move to the next position to repeat the process. in the input before the actual inference begins. The process of prefilling in an LLM can be expressed as cac…

**Figure 5.3.** Figure 5.3: Illustration of the prefilling and decoding processes. In prefilling, the entire input sequence is processed together and the KV cache is filled. In decoding, the LLM generates the output sequence step by step based on the prefilled KV cache. process is memory-bound …

**Figure 5.4.** Figure 5.4: A search tree for decoding. At each node, we expand the tree by considering all possible tokens, each leading to a new node representing a potential continuation of the text. Here we highlight a path through nodes 0, 3, 9, 11, and 17. The path represents the output s…

**Figure 5.5.** Figure 5.5: Illustrations of greedy decoding, beam decoding, top-k decoding and top-p decoding methods (in one decoding step). We can adjust the beam width K to balance search efficiency and accuracy. But a very large beam width might not be helpful. In many practical applicati…

**Figure 5.6.** Figure 5.6: Histogram estimates of the distributions generated by the Softmax function with different values of the temperature parameter β. cumulative probability of the tokens in the pool reaches or exceeds p (we denote the size of the candidate pool at this point as kp). The …

**Figure 5.7.** Figure 5.7: Illustration of one step of speculative decoding. The goal is to predict as many next tokens as possible using the draft model. There are four sub-steps. Given the context, we first use the draft model to predict the next τ tokens. Then, we evaluate these predictions…

**Figure 5.8.** Figure 5.8: Illustrations of basic batching methods. We use a 2D layout to illustrate the batch, where each square represents a token. Red squares indicate tokens in the prefilling stage, blue squares represent tokens in the decoding stage, green squares denote padding tokens, a…

**Figure 5.9.** Figure 5.9: Illustration of the LLM inference architecture involving a scheduler and an inference engine. Each time, the scheduler selects a number of user requests to form a batch and sends it to the inference engine. The scheduler can interact with the inference engine and adj…

**Figure 5.10.** Figure 5.10: Illustrations of request-level scheduling and iteration-based scheduling. In request-level scheduling, once a batch is created and sent to the inference engine, we cannot adjust the batch. In other words, scheduling only occurs after the processing of a batch finish…

**Figure 5.11.** Figure 5.11: Illustration of batch adjustment in continuous batching. Instead of fixing a batch of input sequences and processing them to completion (as in request-level batching), continuous batching dynamically updates the batch during inference. The system continuously accept…

**Figure 5.12.** Figure 5.12: Illustration of memory allocation in PagedAttention. There are two sequences in the batch, as illustrated in sub-figure (a). Since the memory is fragmented, the KV cache is stored in a large unused block of memory in standard self-attention (see sub-figure (b)), but…

**Figure 5.13.** Figure 5.13: Comparison of simple iteration-based scheduling and chunked prefilling. Pxy denotes the y-th prefilling step for sequence x, and Dxy denotes the y-th decoding step for sequence x. In simple iteration-based scheduling (or prefilling-prioritized scheduling), since pre…

**Figure 5.14.** Figure 5.14: Scaling for pre-training, fine-tuning and inference stages [Briski, 2025]. problems, such as complex math problems [Snell et al., 2025]. Unlike pre-training and finetuning scaling, which focuses on improving LLMs via parameter updates, inference-time scaling improv…

**Figure 5.15.** Figure 5.15: Illustration of multi-step reasoning. This example is from the PRM800K dataset [Lightman et al., 2024]. Given a math problem, the LLM is prompted to generate a thinking path (or reasoning path) consisting of several reasoning steps. Each step addresses a sub-problem…

**Figure 5.16.** Figure 5.16: Illustrations of parallel scaling and sequential scaling. In parallel scaling, we obtain multiple solutions by running the LLM several times independently. In sequential scaling, the LLM generates an initial solution. Then, we use the LLM to refine it iteratively, w…

**Figure 5.17.** Figure 5.17: Illustration of step-level search with verifiers. a (j) i = the j-th candidate for the i-th reasoning step, ⊠ = candidate pruned by the LLM’s output probability, and ⊠ = candidate pruned by the verifier. Given the input problem as the root node, we expand the tree b…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs
cs.AI 2025-11 conditional novelty 6.0 of 10

Benign PEFT fine-tuning changes LLM safety and fairness: adapter-based methods (LoRA, IA3) preserve alignment better than prompt-based methods, and the base model strongly moderates outcomes.
Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment
cs.LG 2025-05 conditional novelty 6.0 of 10

SquareχPO, a square-loss variant of χPO, achieves optimal 1/sqrt(n) suboptimality under label privacy and Huber corruption for offline direct alignment with general function classes.
Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details
cs.AI 2026-08 conditional novelty 5.0 of 10

For Other-Play in Yokai, agents trained with different implementation details coordinate across implementations about as well as across seeds, supporting inter-seed cross-play as a proxy for cross-implementation evaluation.
SasAgent: Multi-Agent AI System for Small-Angle Scattering Data Analysis
cs.AI 2025-09 conditional novelty 5.0 of 10

SasAgent connects a large language model to SasView tools through four agents, letting users calculate SLDs, generate synthetic scattering curves, and fit experimental SAS data from text prompts.
Learning to Shop Like Humans: A Review-driven Retrieval-Augmented Recommendation Framework with LLMs
cs.CL 2025-08 conditional novelty 5.0 of 10

RevBrowse retrieves preference-relevant pros and cons from reviews via a contrastively trained module, then uses an LLM to rerank candidates; experiments on four Amazon datasets show consistent improvements over baselines.
Generating Privacy Stories From Software Documentation
cs.SE 2025-06 conditional novelty 5.0 of 10

LLMs can extract privacy behaviors from software documents and draft privacy stories, but the best overall F1 is 0.766, not the abstract's 0.8+.
HEAL: A Hypothesis-Based Preference-Aware Analysis Framework
cs.CL 2025-08 conditional novelty 4.0 of 10

HEAL evaluates preference optimization by measuring ranking accuracy and strength correlation between model likelihoods and proxy reward scores over multi-response hypothesis spaces.
Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Models
cs.CL 2025-07 reject novelty 4.0 of 10

A step-level verifier-guided hybrid of Best-of-N sampling, Monte Carlo tree search, and conditional self-refinement improves reasoning in small instruction-tuned LLMs, claiming up to 28.6-point gains.

Reference graph

Works this paper leans on

296 extracted references · 2 canonical work pages · cited by 8 Pith papers

[1]

write newline shortlabel

" write newline shortlabel "] 0.2em " * write "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format....
[2]

Etc: Encoding long and structured inputs in transformers

[Ainslie et al., 2020] 0.2em Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 268--284, 2020

2020
[3]

Gqa: Training generalized multi-query transformer models from multi-head checkpoints

[Ainslie et al., 2023] 0.2em Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895--4901, 2023

2023
[4]

u rek et al., 2023] 0.2em Ekin Aky \

[Aky \"u rek et al., 2023] 0.2em Ekin Aky \"u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In Proceedings of The Eleventh International Conference on Learning Representations, 2023

2023
[5]

Revisiting neural scaling laws in language and vision

[Alabdulmohsin et al., 2022] 0.2em Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35: 0 22300--22312, 2022

2022
[6]

cosmopedia: how to create large-scale synthetic data for pre-training

[Allal et al., 2024] 0.2em Loubna Ben Allal, Anton Lozhkov, and Daniel van Strien. cosmopedia: how to create large-scale synthetic data for pre-training. https://huggingface.co/blog/cosmopedia, 2024

2024
[7]

The falcon series of open language models

[Almazrouei et al., 2023] 0.2em Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \' e rouane Debbah, \' E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models. arXiv preprint a...

arXiv 2023
[8]

Neural module networks

[Andreas et al., 2016] 0.2em Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39--48, 2016

2016

Show all 296 references

[9]

Unitary evolution recurrent neural networks

[Arjovsky et al., 2016] 0.2em Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pages 1120--1128, 2016

2016
[10]

Situational awareness: The decade ahead, 2024

[Aschenbrenner, 2024] 0.2em Leopold Aschenbrenner. Situational awareness: The decade ahead, 2024. URL https://situational-awareness.ai/

2024
[11]

Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan

[Askell et al., 2021] 0.2em Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Am...

2021 arXiv
[12]

Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V

[Bach et al., 2022] 0.2em Stephen H. Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M. Saiful Bari, Thibault F \' e vry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben - David, Canwen Xu, Gunjan...

2022
[13]

A neural probabilistic language model

[Bengio et al., 2003] 0.2em Yoshua Bengio, R\'ejean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3: 0 1137--1155, 2003

2003
[14]

Greedy layer-wise training of deep networks

[Bengio et al., 2006] 0.2em Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. Advances in neural information processing systems, 19, 2006

2006
[15]

Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atilim Gunes Baydin, Sheila A

[Bengio et al., 2024] 0.2em Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian K. Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atilim Gunes Baydin, Sheila A. McIlrai...

2024
[16]

Pascal recognizing textual entailment challenge (rte-7) at tac 2011

[Bentivogli and Giampiccolo, 2011] 0.2em Luisa Bentivogli and Danilo Giampiccolo. Pascal recognizing textual entailment challenge (rte-7) at tac 2011. https://tac.nist.gov/2011/RTE/, 2011

2011
[17]

Graph of thoughts: Solving elaborate problems with large language models

[Besta et al., 2024] 0.2em Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language...

2024
[18]

Rotary embeddings: A relative revolution

[Biderman et al., 2021] 0.2em Stella Biderman, Sid Black, Charles Foster, Leo Gao, Eric Hallahan, Horace He, Ben Wang, and Phil Wang. Rotary embeddings: A relative revolution. https://blog.eleuther.ai/rotary-embeddings/, 2021

2021
[19]

[Bishop, 2006] 0.2em Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006

2006
[20]

Combining labeled and unlabeled data with co-training

[Blum and Mitchell, 1998] 0.2em Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, pages 92--100, 1998

1998
[21]

[Bradley and Terry, 1952] 0.2em Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

1952
[22]

Reducing transformer key-value cache size with cross-layer attention

[Brandon et al., 2024] 0.2em William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981, 2024

2024 arXiv
[23]

A simple rule-based part of speech tagger

[Brill, 1992] 0.2em Eric Brill. A simple rule-based part of speech tagger. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992, 1992

1992
[24]

Brown, Stephen A

[Brown et al., 1993] 0.2em Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19 0 (2): 0 263--311, 1993

1993
[25]

Language models are few-shot learners

[Brown et al., 2020] 0.2em Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh...

2020
[26]

Lundberg, Harsha Nori, Hamid Palangi, Marco T \' u lio Ribeiro, and Yi Zhang

[Bubeck et al., 2023] 0.2em S \' e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco T \' u lio Ribeiro, and Yi Zhang. Sparks of artificial gener...

2023 arXiv
[27]

Recurrent memory transformer

[Bulatov et al., 2022] 0.2em Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35: 0 11079--11091, 2022

2022
[28]

Learning to rank using gradient descent

[Burges et al., 2005] 0.2em Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89--96, 2005

2005
[29]

Weak-to-strong generalization: Eliciting strong capabilities with weak supervision

[Burns et al., 2023] 0.2em Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with we...

2023 arXiv
[30]

Weak-to-strong generalization, 2023 b

[Burns et al., 2023] 0.2em Collin Burns, Jan Leike, Leopold Aschenbrenner, Jeffrey Wu, Pavel Izmailov, Leo Gao, Bowen Baker, and Jan Hendrik Kirchner. Weak-to-strong generalization, 2023 b . URL https://https://openai.com/index/weak-to-strong-generalization

2023
[31]

Broken neural scaling laws

[Caballero et al., 2023] 0.2em Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023

2023
[32]

Learning to rank: from pairwise approach to listwise approach

[Cao et al., 2007] 0.2em Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pages 129--136, 2007

2007
[33]

Efficient prompting methods for large language models: A survey

[Chang et al., 2024] 0.2em Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Tong Xiao, and Jingbo Zhu. Efficient prompting methods for large language models: A survey. arXiv preprint arXiv:2404.01077, 2024

2024 arXiv
[34]

Statistical parsing with a context-free grammar and word statistics

[Charniak, 1997] 0.2em Eugene Charniak. Statistical parsing with a context-free grammar and word statistics. AAAI/IAAI, 2005 0 (598-603): 0 18, 1997

1997
[35]

Unleashing the potential of prompt engineering in large language models: a comprehensive review

[Chen et al., 2023] 0.2em Banghao Chen, Zhaofeng Zhang, Nicolas Langren \'e , and Shengxin Zhu. Unleashing the potential of prompt engineering in large language models: a comprehensive review. arXiv preprint arXiv:2310.14735, 2023 a

2023 arXiv
[36]

Alpagasus: Training a better alpaca with fewer data

[Chen et al., 2023] 0.2em Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701, 2023 b

2023 arXiv
[37]

Alpagasus: Training a better alpaca with fewer data

[Chen et al., 2024] 0.2em Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning...

2024
[38]

Extending context window of large language models via positional interpolation

[Chen et al., 2023] 0.2em Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023 c

2023 arXiv
[39]

The lottery ticket hypothesis for pre-trained bert networks

[Chen et al., 2020] 0.2em Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33: 0 15834--15846, 2020

2020
[40]

Self-play fine-tuning converts weak language models to strong language models

[Chen et al., 2024] 0.2em Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024 b

2024 arXiv
[41]

Adapting language models to compress contexts

[Chevalier et al., 2023] 0.2em Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3829--3846, 2023

2023
[42]

Kerple: Kernelized relative positional embedding for length extrapolation

[Chi et al., 2022] 0.2em Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35: 0 8386--8399, 2022

2022
[43]

Dissecting transformer length extrapolation via the lens of receptive field analysis

[Chi et al., 2023] 0.2em Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume ...

2023
[44]

Gonzalez, Ion Stoica, and Eric P

[Chiang et al., 2023] 0.2em Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90\ quality, March 2023. URL...

2023
[45]

[Chowdhery et al., 2022] 0.2em Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Bar...

2022 arXiv
[46]

Deep reinforcement learning from human preferences

[Christiano et al., 2017] 0.2em Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

2017
[47]

A survey of chain of thought reasoning: Advances, frontiers and future

[Chu et al., 2023] 0.2em Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402, 2023

2023 arXiv
[48]

[Chung et al., 2022] 0.2em Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Nar...

2022 arXiv
[49]

Electra: Pre-training text encoders as discriminators rather than generators

[Clark et al., 2019] 0.2em Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. In Proceedings of International Conference on Learning Representations, 2019

2019
[50]

Training verifiers to solve math word problems

[Cobbe et al., 2021] 0.2em Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprin...

2021 arXiv
[51]

Unsupervised cross-lingual representation learning at scale

[Conneau et al., 2020] 0.2em Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, \'E douard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceed...

2020
[52]

Reward model ensembles help mitigate overoptimization

[Coste et al., 2024] 0.2em Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024

2024
[53]

ULTRAFEEDBACK : Boosting language models with scaled AI feedback

[Cui et al., 2024] 0.2em Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK : Boosting language models with scaled AI feedback. In Proceedings of the 41st Internation...

2024
[54]

Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers

[Dai et al., 2023] 0.2em Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics: ACL 2023, ...

2023
[55]

Transformer-xl: Attentive language models beyond a fixed-length context

[Dai et al., 2019] 0.2em Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguist...

2019
[56]

Flashattention: Fast and memory-efficient exact attention with io-awareness

[Dao et al., 2022] 0.2em Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

2022
[57]

Universal transformers

[Dehghani et al., 2018] 0.2em Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and ukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819, 2018

2018 arXiv
[58]

Language modeling is compression

[Deletang et al., 2024] 0.2em Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression. In The...

2024
[59]

Rlprompt: Optimizing discrete text prompts with reinforcement learning

[Deng et al., 2022] 0.2em Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in ...

2022
[60]

Bert: Pre-training of deep bidirectional transformers for language understanding

[Devlin et al., 2019] 0.2em Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computatio...

2019
[61]

Longrope: Extending llm context window beyond 2 million tokens

[Ding et al., 2024] 0.2em Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753, 2024

2024 arXiv
[62]

Automatically constructing a corpus of sentential paraphrases

[Dolan and Brockett, 2005] 0.2em Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of Third International Workshop on Paraphrasing (IWP2005), 2005

2005
[63]

Unified language model pre-training for natural language understanding and generation

[Dong et al., 2019] 0.2em Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32, 2019

2019
[64]

A survey on in-context learning

[Dong et al., 2022] 0.2em Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022

2022 arXiv
[65]

Attention is not all you need: Pure attention loses rank doubly exponentially with depth

[Dong et al., 2021] 0.2em Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793--2803. PMLR, 2021

2021
[66]

a rli, Ekin Aky \

[Drozdov et al., 2022] 0.2em Andrew Drozdov, Nathanael Sch \"a rli, Ekin Aky \"u rek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. In Proceedings of The Eleventh International Conference ...

2022
[67]

Successive prompting for decomposing complex questions

[Dua et al., 2022] 0.2em Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. Successive prompting for decomposing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1251--1265, 2022

2022
[68]

The llama 3 herd of models

[Dubey et al., 2024] 0.2em Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[69]

Alpacafarm: A simulation framework for methods that learn from human feedback

[Dubois et al., 2024] 0.2em Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Informa...

2024
[70]

Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking

[Eisenstein et al., 2023] 0.2em Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, and Peter Shaw. Helping or herding? reward model ensembles mitigate but do not eliminate...

2023 arXiv
[71]

Neural architecture search: A survey

[Elsken et al., 2019] 0.2em Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20 0 (55): 0 1--21, 2019

2019
[72]

Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201--208, 2010

[Erhan et al., 2010] 0.2em Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201--208, 2010

2010
[73]

Reducing transformer depth on demand with structured dropout

[Fan et al., 2019] 0.2em Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In Proceedings of International Conference on Learning Representations, 2019

2019
[74]

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

[Fedus et al., 2022] 0.2em William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23 0 (1): 0 5232--5270, 2022

2022
[75]

[Fernandes et al., 2023] 0.2em Patrick Fernandes, Aman Madaan, Emmy Liu, Ant \' o nio Farinhas, Pedro Henrique Martins, Amanda Bertsch, Jos \' e G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and Andr \' e F. T. Martins. Bridging the gap: A survey on integrating (h...

2023
[76]

Is it an agent, or just a program?: A taxonomy for autonomous agents

[Franklin and Graesser, 1996] 0.2em Stan Franklin and Art Graesser. Is it an agent, or just a program?: A taxonomy for autonomous agents. In International workshop on agent theories, architectures, and languages, pages 21--35. Springer, 1996

1996
[77]

Complex problem solving: The European perspective

[Frensch and Funke, 2014] 0.2em Peter A Frensch and Joachim Funke. Complex problem solving: The European perspective. Psychology Press, 2014

2014
[78]

The state of sparsity in deep neural networks

[Gale et al., 2019] 0.2em Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019

2019 arXiv
[79]

[Ganguli et al., 2023] 0.2em Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamile Lukosiute, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran - Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jar...

2023 arXiv
[80]

Scaling laws for reward model overoptimization

[Gao et al., 2023] 0.2em Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835--10866. PMLR, 2023 a

2023
[81]

Pal: Program-aided language models

[Gao et al., 2023] 0.2em Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764--10799. PMLR, 2023 b

2023
[82]

Retrieval-augmented generation for large language models: A survey

[Gao et al., 2023] 0.2em Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023 c

2023 arXiv
[83]

What can transformers learn in-context? a case study of simple function classes

[Garg et al., 2022] 0.2em Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35: 0 30583--30598, 2022

2022
[84]

Clustering and ranking: Diversity-preserved instruction selection through expert-aligned quality estimation

[Ge et al., 2024] 0.2em Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Hongxia Ma, Li Zhang, Boxing Chen, Hao Yang, Bei Li, Tong Xiao, and Jingbo Zhu. Clustering and ranking: Diversity-preserved instruction selection through expert-aligned quality estimati...

2024 arXiv
[85]

Gemma: Open Models Based on Gemini Research and Technology , 2024

[Gemma Team, 2024] 0.2em Google DeepMind Gemma Team. Gemma: Open Models Based on Gemini Research and Technology , 2024

2024
[86]

Problems of monetary management: the UK experience

[Goodhart, 1984] 0.2em Charles AE Goodhart. Problems of monetary management: the UK experience. Springer, 1984

1984
[87]

Data and parameter scaling laws for neural machine translation

[Gordon et al., 2021] 0.2em Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5915--5922, 2021

2021
[88]

Mamba: Linear-time sequence modeling with selective state spaces

[Gu and Dao, 2023] 0.2em Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

2023 arXiv
[89]

Textbooks are all you need

[Gunasekar et al., 2023] 0.2em Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \' e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, S \' e bastien ...

2023 arXiv
[90]

Connecting large language models with evolutionary algorithms yields powerful prompt optimizers

[Guo et al., 2024] 0.2em Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learn...

2024
[91]

Gmat: Global memory augmentation for transformers

[Gupta and Berant, 2020] 0.2em Ankit Gupta and Jonathan Berant. Gmat: Global memory augmentation for transformers. arXiv preprint arXiv:2006.03274, 2020

2020 arXiv
[92]

Memory-efficient transformers via top-k attention

[Gupta et al., 2021] 0.2em Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. Memory-efficient transformers via top-k attention. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pages 39--52, 2021

2021
[93]

Pre-trained models: Past, present and future

[Han et al., 2021] 0.2em Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, an...

2021
[94]

Parameter-efficient fine-tuning for large models: A comprehensive survey

[Han et al., 2024] 0.2em Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608, 2024

2024 arXiv
[95]

Pipedream: Fast and efficient pipeline parallel dnn training

[Harlap et al., 2018] 0.2em Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018

2018 arXiv
[96]

Rethinking imagenet pre-training

[He et al., 2019] 0.2em Kaiming He, Ross Girshick, and Piotr Doll \'a r. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4918--4927, 2019

2019
[97]

Deberta: Decoding-enhanced bert with disentangled attention

[He et al., 2021] 0.2em Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In Proceedings of International Conference on Learning Representations, 2021

2021
[98]

Gaussian error linear units (gelus)

[Hendrycks and Gimpel, 2016] 0.2em Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

2016 arXiv
[99]

Pretrained transformers improve out-of-distribution robustness

[Hendrycks et al., 2020] 0.2em Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,...

2020
[100]

Measuring massive multitask language understanding

[Hendrycks et al., 2021] 0.2em Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of International Conference on Learning Representations, 2021

2021
[101]

Deep learning scaling is predictable, empirically

[Hestness et al., 2017] 0.2em Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

2017 arXiv
[102]

Instruction following without instruction tuning, 2024

[Hewitt, 2024] 0.2em John Hewitt. Instruction following without instruction tuning, 2024. URL https://nlp.stanford.edu/ johnhew/instruction-following.html

2024
[103]

Instruction following without instruction tuning

[Hewitt et al., 2024] 0.2em John Hewitt, Nelson F Liu, Percy Liang, and Christopher D Manning. Instruction following without instruction tuning. arXiv preprint arXiv:2409.14254, 2024

2024 arXiv
[104]

Long short-term memory

[Hochreiter and Schmidhuber, 1997] 0.2em Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

1997
[105]

Rae, Oriol Vinyals, and Laurent Sifre

[Hoffmann et al., 2022] 0.2em Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan ...

2022 arXiv
[106]

Unnatural instructions: Tuning language models with (almost) no human labor

[Honovich et al., 2023] 0.2em Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

2023
[107]

Parameter-efficient transfer learning for NLP

[Houlsby et al., 2019] 0.2em Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP . In Proceedings of the 36th International Conference on M...

2019
[108]

Lo RA : Low-rank adaptation of large language models

[Hu et al., 2022] 0.2em Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

2022
[109]

Dynamic programming-based search algorithms in NLP

[Huang, 2009] 0.2em Liang Huang. Dynamic programming-based search algorithms in NLP . In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North A merican Chapter of the Association for Computational Linguistics, Companion Volume: Tutorial Abstracts , 2009

2009
[110]

Gpipe: Efficient training of giant neural networks using pipeline parallelism

[Huang et al., 2019] 0.2em Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural ...

2019
[111]

Block-recurrent transformers

[Hutchins et al., 2022] 0.2em DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block-recurrent transformers. Advances in neural information processing systems, 35: 0 33248--33261, 2022

2022
[112]

Statistical methods for speech recognition

[Jelinek, 1998] 0.2em Frederick Jelinek. Statistical methods for speech recognition. MIT Press, 1998

1998
[113]

Mistral 7b

[Jiang et al., 2023] 0.2em Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \'e lio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le...

2023 arXiv
[114]

Llmlingua: Compressing prompts for accelerated inference of large language models

[Jiang et al., 2023] 0.2em Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1...

2023
[115]

How can we know what language models know? Transactions of the Association for Computational Linguistics, 8: 0 423--438, 2020

[Jiang et al., 2020] 0.2em Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8: 0 423--438, 2020

2020
[116]

Tinybert: Distilling bert for natural language understanding

[Jiao et al., 2020] 0.2em Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163--4174, 2020

2020
[117]

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

[Joshi et al., 2017] 0.2em Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volu...

2017
[118]

Spanbert: Improving pre-training by representing and predicting spans

[Joshi et al., 2020] 0.2em Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8: 0 64--77, 2020

2020
[119]

[Jurafsky and Martin, 2008] 0.2em Dan Jurafsky and James H. Martin. Speech and Language Processing (2nd ed.). Prentice Hall, 2008

2008
[120]

Thinking, fast and slow

[Kahneman, 2011] 0.2em Daniel Kahneman. Thinking, fast and slow. macmillan, 2011

2011
[121]

Scaling laws for neural language models

[Kaplan et al., 2020] 0.2em Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

2020 arXiv
[122]

Transformers are rnns: Fast autoregressive transformers with linear attention

[Katharopoulos et al., 2020] 0.2em Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156--5165. PMLR, 2020

2020
[123]

Generalization through memorization: Nearest neighbor language models

[Khandelwal et al., 2020] 0.2em Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020

2020
[124]

Decomposed prompting: A modular approach for solving complex tasks

[Khot et al., 2023] 0.2em Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In Proceedings of The Eleventh International Conference on Learning Represen...

2023
[125]

Mahoney, Yakun Sophia Shao, and Amir Gholami

[Kim et al., 2023] 0.2em Sehoon Kim, Coleman Hooper, Thanakul Wattanawong, Minwoo Kang, Ruohan Yan, Hasan Genc, Grace Dinh, Qijing Huang, Kurt Keutzer, Michael W. Mahoney, Yakun Sophia Shao, and Amir Gholami. Full stack optimization of transformer inference: a survey. arXiv pr...

2023 arXiv
[126]

Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

[Kirkpatrick et al., 2017] 0.2em James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Ove...

2017
[127]

Statistical Machine Translation

[Koehn, 2010] 0.2em Philipp Koehn. Statistical Machine Translation. Cambridge University Press, 2010

2010
[128]

Large language models are zero-shot reasoners

[Kojima et al., 2022] 0.2em Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 0 22199--22213, 2022

2022
[129]

Reducing activation recomputation in large transformer models

[Korthikanti et al., 2023] 0.2em Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023

2023
[130]

Specification gaming: the flip side of ai ingenuity

[Krakovna et al., 2020] 0.2em Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingenuity. https://deepmind.google/discover/blog/specification-gaming-...

2020
[131]

Do models really learn to follow instructions? an empirical study of instruction tuning

[Kung and Peng, 2023] 0.2em Po-Nien Kung and Nanyun Peng. Do models really learn to follow instructions? an empirical study of instruction tuning. arXiv preprint arXiv:2305.11383, 2023

2023 arXiv
[132]

Efficient memory management for large language model serving with pagedattention

[Kwon et al., 2023] 0.2em Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. arXiv preprint arXiv:2309.06180, 2023

2023 arXiv
[133]

Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

[Lake and Baroni, 2018] 0.2em Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, pages 2873--2882. PMLR, 2018

2018
[134]

Smith, and Hannaneh Hajishirzi

[Lambert et al., 2024] 0.2em Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. ar...

2024 arXiv
[135]

Cross-lingual language model pretraining

[Lample and Conneau, 2019] 0.2em Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019

2019 arXiv
[136]

Albert: A lite bert for self-supervised learning of language representations

[Lan et al., 2020] 0.2em Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In Proceedings of International Conference on Learning Representations, 2020

2020
[137]

Rlaif: Scaling reinforcement learning from human feedback with ai feedback

[Lee et al., 2023] 0.2em Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:...

2023 arXiv
[138]

The power of scale for parameter-efficient prompt tuning

[Lester et al., 2021] 0.2em Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045--3059, 2021

2021
[139]

Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

[Lewis et al., 2020] 0.2em Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In P...

2020
[140]

Deliberate then generate: Enhanced prompting framework for text generation

[Li et al., 2023] 0.2em Bei Li, Rui Wang, Junliang Guo, Kaitao Song, Xu Tan, Hany Hassan, Arul Menezes, Tong Xiao, Jiang Bian, and JingBo Zhu. Deliberate then generate: Enhanced prompting framework for text generation. arXiv preprint arXiv:2305.19835, 2023 a

2023 arXiv
[141]

Learning to Rank for Information Retrieval and Natural Language Processing

[Li, 2011] 0.2em Hang Li. Learning to Rank for Information Retrieval and Natural Language Processing. Online access: Morgan & Claypool Synthesis Collection Five. Morgan & Claypool Publishers, 2011. ISBN 9781608457076

2011
[142]

A survey on retrieval-augmented text generation

[Li et al., 2022] 0.2em Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. A survey on retrieval-augmented text generation. arXiv preprint arXiv:2202.01110, 2022

2022 arXiv
[143]

Functional interpolation for relative positions improves long context transformers

[Li et al., 2024] 0.2em Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. In The Twelft...

2024
[144]

Sequence parallelism: Long sequence training from system perspective

[Li et al., 2023] 0.2em Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

2023
[145]

Prefix-tuning: Optimizing continuous prompts for generation

[Li and Liang, 2021] 0.2em Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro...

2021
[146]

A practical survey on zero-shot prompt design for in-context learning

[Li, 2023] 0.2em Yinheng Li. A practical survey on zero-shot prompt design for in-context learning. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 641--647, 2023

2023
[147]

Compressing context to enhance inference efficiency of large language models

[Li et al., 2023] 0.2em Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342--6353, 2023 c

2023
[148]

Scaling down to scale up: A guide to parameter-efficient fine-tuning

[Lialin et al., 2023] 0.2em Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv preprint arXiv:2303.15647, 2023

2023 arXiv
[149]

Let's verify step by step

[Lightman et al., 2024] 0.2em Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024

2024
[150]

Deepseek-v3 technical report

[Liu et al., 2024] 0.2em Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 a

2024 arXiv
[151]

[Liu et al., 2022] 0.2em Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration...

2022
[152]

Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

[Liu et al., 2023] 0.2em Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55 0 (9): 0 1--35, 2023 a

2023
[153]

Statistical rejection sampling improves preference optimization

[Liu et al., 2024] 0.2em Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024 b

2024
[154]

Learning to rank for information retrieval

[Liu, 2009] 0.2em Tie-Yan Liu. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval , 3 0 (3): 0 225--331, 2009

2009
[155]

Gpt understands, too

[Liu et al., 2023] 0.2em Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open, 2023 b

2023
[156]

Prompting frameworks for large language models: A survey

[Liu et al., 2023] 0.2em Xiaoxia Liu, Jingyi Wang, Jun Sun, Xiaohan Yuan, Guoliang Dong, Peng Di, Wenhai Wang, and Dongxia Wang. Prompting frameworks for large language models: A survey. arXiv preprint arXiv:2311.12785, 2023 c

2023 arXiv
[157]

Forgetting curve: A reliable method for evaluating memorization capability for long-context models

[Liu et al., 2024] 0.2em Xinyu Liu, Runsong Zhao, Pengcheng Huang, Chunyang Xiao, Bei Li, Jingang Wang, Tong Xiao, and Jingbo Zhu. Forgetting curve: A reliable method for evaluating memorization capability for long-context models. In Proceedings of the 2024 Conference on Empir...

2024
[158]

Roberta: A robustly optimized bert pretraining approach

[Liu et al., 2019] 0.2em Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

2019 arXiv
[159]

Le, Barret Zoph, Jason Wei, and Adam Roberts

[Longpre et al., 2023] 0.2em Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In International Conference o...

2023
[160]

Mega: Moving average equipped gated attention

[Ma et al., 2023] 0.2em Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In The Eleventh International Conference on Learning Representations, 2023

2023
[161]

Megalodon: Efficient llm pretraining and inference with unlimited context length

[Ma et al., 2024] 0.2em Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer, Omer Levy, and Chunting Zhou. Megalodon: Efficient llm pretraining and inference with unlimited context length. arXiv preprint arXiv:2404.08801, 2024

2024 arXiv
[162]

Self-refine: Iterative refinement with self-feedback

[Madaan et al., 2024] 0.2em Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter ...

2024
[163]

Human language understanding & reasoning

[Manning, 2022] 0.2em Christopher D Manning. Human language understanding & reasoning. Daedalus, 151 0 (2): 0 127--138, 2022

2022
[164]

Negative evidence in language acquisition

[Marcus, 1993] 0.2em Gary F Marcus. Negative evidence in language acquisition. Cognition, 46 0 (1): 0 53--85, 1993

1993
[165]

-former: Infinite memory transformer-former: Infinite memory transformer

[Martins et al., 2022] 0.2em Pedro Henrique Martins, Zita Marinho, and Andr \'e FT Martins. -former: Infinite memory transformer-former: Infinite memory transformer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

2022
[166]

Multi-hop question answering

[Mavi et al., 2024] 0.2em Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. Multi-hop question answering. Foundations and Trends in Information Retrieval , 17 0 (5): 0 457--586, 2024

2024
[167]

Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019

[Michel et al., 2019] 0.2em Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019

2019
[168]

Mixed precision training

[Micikevicius et al., 2018] 0.2em Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In Proceedings of International Conference ...

2018
[169]

Nonlinear multiobjective optimization, volume 12

[Miettinen, 1999] 0.2em Kaisa Miettinen. Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media, 1999

1999
[170]

Efficient estimation of word representations in vector space

[Mikolov et al., 2013] 0.2em Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR 2013), 2013 a

2013
[171]

Distributed representations of words and phrases and their compositionality

[Mikolov et al., 2013] 0.2em Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Vol...

2013
[172]

Multi-hop reading comprehension through question decomposition and rescoring

[Min et al., 2019] 0.2em Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop reading comprehension through question decomposition and rescoring. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6097--6109, 2019

2019
[173]

Large language models: A survey

[Minaee et al., 2024] 0.2em Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024

2024 arXiv
[174]

Cross-task generalization via natural language crowdsourcing instructions

[Mishra et al., 2022] 0.2em Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: ...

2022
[175]

Asynchronous methods for deep reinforcement learning

[Mnih et al., 2016] 0.2em Volodymyr Mnih, Adri \`a Puigdom \`e nech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P Lillicrap, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference ...

2016
[176]

Random-access infinite context length for transformers

[Mohtashami and Jaggi, 2024] 0.2em Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. Advances in Neural Information Processing Systems, 36, 2024

2024
[177]

Learning to compress prompts with gist tokens

[Mu et al., 2024] 0.2em Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36, 2024

2024
[178]

Leave no context behind: Efficient infinite context transformers with infini-attention

[Munkhdalai et al., 2024] 0.2em Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07143, 2024

2024 arXiv
[179]

Webgpt: Browser-assisted question-answering with human feedback

[Nakano et al., 2021] 0.2em Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Ch...

2021 arXiv
[180]

Efficient large-scale language model training on gpu clusters using megatron-lm

[Narayanan et al., 2021] 0.2em Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language m...

2021
[181]

Policy invariance under reward transformations: Theory and application to reward shaping

[Ng et al., 1999] 0.2em Andrew Y Ng, Daishi Harada, and Stuart J Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, pages 278--287, 1999

1999
[182]

Learning to reason with llms, September 2024

[OpenAI, 2024] 0.2em OpenAI. Learning to reason with llms, September 2024. URL https://openai.com/index/learning-to-reason-with-llms/

2024
[183]

[Ouyang et al., 2022] 0.2em Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, ...

2022
[184]

Future lens: Anticipating subsequent tokens from a single hidden state

[Pal et al., 2023] 0.2em Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C Wallace, and David Bau. Future lens: Anticipating subsequent tokens from a single hidden state. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 548--560, 2023

2023
[185]

The effects of reward misspecification: Mapping and mitigating misaligned models

[Pan et al., 2022] 0.2em Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022

2022
[186]

Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies

[Pan et al., 2024] 0.2em Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computati...

2024
[187]

Talm: Tool augmented language models

[Parisi et al., 2022] 0.2em Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022

2022 arXiv
[188]

Continual lifelong learning with neural networks: A review

[Parisi et al., 2019] 0.2em German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113: 0 54--71, 2019

2019
[189]

Image transformer

[Parmar et al., 2018] 0.2em Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pages 4055--4064. PMLR, 2018

2018
[190]

The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

[Penedo et al., 2023] 0.2em Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data,...

2023 arXiv
[191]

Ya RN : Efficient context window extension of large language models

[Peng et al., 2024] 0.2em Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Ya RN : Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024

2024
[192]

[Pennington et al., 2014] 0.2em Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543, 2014

2014
[193]

Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

[Peters et al., 2018] 0.2em Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Co...

2018
[194]

The analysis of permutations

[Plackett, 1975] 0.2em Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24 0 (2): 0 193--202, 1975

1975
[195]

Grips: Gradient-free, edit-based instruction search for prompting large language models

[Prasad et al., 2023] 0.2em Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based instruction search for prompting large language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin...

2023
[196]

Train short, test long: Attention with linear biases enables input length extrapolation

[Press et al., 2022] 0.2em Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In Proceedings of International Conference on Learning Representations, 2022

2022
[197]

Measuring and narrowing the compositionality gap in language models

[Press et al., 2023] 0.2em Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687--5711, 2023

2023
[198]

gradient descent

[Pryzant et al., 2023] 0.2em Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with "gradient descent" and beam search. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

2023
[199]

Pre-trained models for natural language processing: A survey

[Qiu et al., 2020] 0.2em Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63 0 (10): 0 1872--1897, 2020

2020
[200]

Improving language understanding by generative pre-training

[Radford et al., 2018] 0.2em Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. OpenAI, 2018

2018
[201]

Language models are unsupervised multitask learners

[Radford et al., 2019] 0.2em Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8), 2019

2019
[202]

Learning transferable visual models from natural language supervision

[Radford et al., 2021] 0.2em Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supe...

2021
[203]

Compressive transformers for long-range sequence modelling

[Rae et al., 2019] 0.2em Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2019

2019
[204]

Direct preference optimization: Your language model is secretly a reward model

[Rafailov et al., 2024] 0.2em Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

2024
[205]

[Raffel et al., 2020] 0.2em Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21...

2020
[206]

Searching for activation functions

[Ramachandran et al., 2017] 0.2em Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017

2017 arXiv
[207]

Experience replay for continual learning

[Rolnick et al., 2019] 0.2em David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019

2019
[208]

A constructive prediction of the generalization error across scales

[Rosenfeld et al., 2020] 0.2em Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In Proceedings of International Conference on Learning Representations, 2020

2020
[209]

A survey of llm surveys

[Ruan et al., 2024] 0.2em Junhao Ruan, Long Meng, Weiqiao Shan, Tong Xiao, and Jingbo Zhu. A survey of llm surveys. https://github.com/NiuTrans/ABigSurveyOfLLMs, 2024

2024
[210]

Learning to retrieve prompts for in-context learning

[Rubin et al., 2022] 0.2em Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,...

2022
[211]

Human Compatible: Artificial Intelligence and the Problem of Controls

[Russell, 2019] 0.2em Stuart Russell. Human Compatible: Artificial Intelligence and the Problem of Controls. Viking, 2019

2019
[212]

Movement pruning: Adaptive sparsity by fine-tuning

[Sanh et al., 2020] 0.2em Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems, 33: 0 20378--20389, 2020

2020
[213]

Multitask prompted training enables zero-shot task generalization

[Sanh et al., 2022] 0.2em Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhab...

2022
[214]

Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel

[Schick et al., 2023] 0.2em Timo Schick, Jane A. Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. PEER : A collaborative language model. In Proceedings of The Eleventh International Co...

2023
[215]

Toolformer: Language models can teach themselves to use tools

[Schick et al., 2024] 0.2em Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Process...

2024
[216]

Deep learning in neural networks: An overview

[Schmidhuber, 2015] 0.2em J \"u rgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61: 0 85--117, 2015

2015
[217]

Trust region policy optimization

[Schulman et al., 2015] 0.2em John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37, pages 1889--1897, 2015

2015
[218]

Proximal policy optimization algorithms

[Schulman et al., 2017] 0.2em John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[219]

Improving neural machine translation models with monolingual data

[Sennrich et al., 2016] 0.2em Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86--96, 2016

2016
[220]

Bidirectional attention flow for machine comprehension

[Seo et al., 2017] 0.2em Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In Proceedings of International Conference on Learning Representations, 2017

2017
[221]

Prediction and entropy of printed english

[Shannon, 1951] 0.2em Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal, 30 0 (1): 0 50--64, 1951

1951
[222]

Self-attention with relative position representations

[Shaw et al., 2018] 0.2em Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ...

2018
[223]

Fast transformer decoding: One write-head is all you need

[Shazeer, 2019] 0.2em Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

2019 arXiv
[224]

Glu variants improve transformer

[Shazeer, 2020] 0.2em Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

2020 arXiv
[225]

Q-bert: Hessian based ultra low precision quantization of bert

[Shen et al., 2020] 0.2em Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pa...

2020
[226]

Megatron-lm: Training multi-billion parameter language models using model parallelism

[Shoeybi et al., 2019] 0.2em Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

2019 arXiv
[227]

Defining and characterizing reward gaming

[Skalse et al., 2022] 0.2em Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35: 0 9460--9471, 2022

2022
[228]

Learning by distilling context

[Snell et al., 2022] 0.2em Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv preprint arXiv:2209.15189, 2022

2022 arXiv
[229]

Recursive deep models for semantic compositionality over a sentiment treebank

[Socher et al., 2013] 0.2em Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical metho...

2013
[230]

Mass: Masked sequence to sequence pre-training for language generation

[Song et al., 2019] 0.2em Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, pages 5926--5936. PMLR, 2019

2019
[231]

Learning to summarize with human feedback

[Stiennon et al., 2020] 0.2em Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33: 0 3008--3021, 2020

2020
[232]

Roformer: Enhanced transformer with rotary position embedding

[Su et al., 2024] 0.2em Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

2024
[233]

Mobilebert: a compact task-agnostic bert for resource-limited devices

[Sun et al., 2020] 0.2em Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 215...

2020
[234]

Sequence to sequence learning with neural networks

[Sutskever et al., 2014] 0.2em Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014

2014
[235]

Sutton and Andrew G

[Sutton and Barto, 2018] 0.2em Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction (2nd ed.). The MIT Press, 2018

2018
[236]

Algorithms for reinforcement learning

[Szepesv \'a ri, 2010] 0.2em Csaba Szepesv \'a ri. Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 4 0 (1): 0 1--103, 2010

2010
[237]

The web as a knowledge-base for answering complex questions

[Talmor and Berant, 2018] 0.2em Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. arXiv preprint arXiv:1803.06643, 2018

2018 arXiv
[238]

Hashimoto

[Taori et al., 2023] 0.2em Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

2023
[239]

Efficient transformers: A survey

[Tay et al., 2020] 0.2em Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. CoRR, abs/2009.06732, 2020

2020 arXiv
[240]

Gemma 2: Improving open language models at a practical size

[Team et al., 2024] 0.2em Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. Gemma 2: Improving open language models at a practical size. arXiv pr...

2024 arXiv
[241]

Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

[Teknium, 2023] 0.2em Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023. URL https://huggingface.co/datasets/teknium/OpenHermes-2.5

2023
[242]

Llama: Open and efficient foundation language models

[Touvron et al., 2023] 0.2em Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: O...

2023 arXiv
[243]

Llama 2: Open foundation and fine-tuned chat models

[Touvron et al., 2023] 0.2em Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu,...

2023 arXiv
[244]

Solving math word problems with process-and outcome-based feedback

[Uesato et al., 2022] 0.2em Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022

2022 arXiv
[245]

Attention is all you need

[Vaswani et al., 2017] 0.2em Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems, volume 30, 2017

2017
[246]

Transformers learn in-context by gradient descent

[Von Oswald et al., 2023] 0.2em Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo \ a o Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In Proceedings of International Conference on Machine Le...

2023
[247]

Esrl: Efficient sampling-based reinforcement learning for sequence generation

[Wang et al., 2024] 0.2em Chenglong Wang, Hang Zhou, Yimin Hu, Yifu Huo, Bei Li, Tongran Liu, Tong Xiao, and Jingbo Zhu. Esrl: Efficient sampling-based reinforcement learning for sequence generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 19107...

2024
[248]

A comprehensive survey of continual learning: Theory, method and application

[Wang et al., 2023] 0.2em Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023 a

2023 arXiv
[249]

Learning deep transformer models for machine translation

[Wang et al., 2019] 0.2em Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810-...

2019
[250]

Rationale-augmented ensembles in language models

[Wang et al., 2022] 0.2em Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Rationale-augmented ensembles in language models. arXiv preprint arXiv:2207.00747, 2022 a

2022 arXiv
[251]

Self-consistency improves chain of thought reasoning in language models

[Wang et al., 2023] 0.2em Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In Proceedings of The Eleventh International Conference on Learn...

2023
[252]

Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

[Wang et al., 2022] 0.2em Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mond...

2022
[253]

Smith, Iz Beltagy, and Hannaneh Hajishirzi

[Wang et al., 2023] 0.2em Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open...

2023
[254]

Self-instruct: Aligning language models with self-generated instructions

[Wang et al., 2023] 0.2em Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for ...

2023
[255]

A comprehensive survey of forgetting in deep learning beyond continual learning

[Wang et al., 2023] 0.2em Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. arXiv preprint arXiv:2307.09218, 2023 e

2023 arXiv
[256]

Neural network acceptability judgments

[Warstadt et al., 2019] 0.2em Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7: 0 625--641, 2019

2019
[257]

Finetuned language models are zero-shot learners

[Wei et al., 2022] 0.2em Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In Proceedings of International Conference on Learning Representations, 2022 a

2022
[258]

Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

[Wei et al., 2022] 0.2em Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of...

2022 arXiv
[259]

Chi, Quoc V

[Wei et al., 2022] 0.2em Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 0 24824-...

2022
[260]

Generating sequences by learning to self-correct

[Welleck et al., 2023] 0.2em Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In Proceedings of The Eleventh International Conference on Learning Representations, 2023

2023
[261]

How to train really large models on many gpus? lilianweng.github.io, Sep 2021

[Weng, 2021] 0.2em Lilian Weng. How to train really large models on many gpus? lilianweng.github.io, Sep 2021. URL https://lilianweng.github.io/posts/2021-09-25-train-large/

2021
[262]

Some moral and technical consequences of automation: As machines learn they may develop unforeseen strategies at rates that baffle their programmers

[Wiener, 1960] 0.2em Norbert Wiener. Some moral and technical consequences of automation: As machines learn they may develop unforeseen strategies at rates that baffle their programmers. Science, 131 0 (3410): 0 1355--1358, 1960

1960
[263]

A broad-coverage challenge corpus for sentence understanding through inference

[Williams et al., 2018] 0.2em Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:...

2018
[264]

Simple statistical gradient-following algorithms for connectionist reinforcement learning

[Williams, 1992] 0.2em Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8: 0 229--256, 1992

1992
[265]

Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models

[Wingate et al., 2022] 0.2em David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 562...

2022
[266]

Do language models plan for future tokens? arXiv preprint arXiv:2404.00859, 2024

[Wu et al., 2024] 0.2em Wilson Wu, John X Morris, and Lionel Levine. Do language models plan for future tokens? arXiv preprint arXiv:2404.00859, 2024

2024 arXiv
[267]

Memorizing transformers

[Wu et al., 2021] 0.2em Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In Proceedings of International Conference on Learning Representations, 2021

2021
[268]

Smith, Mari Ostendorf, and Hannaneh Hajishirzi

[Wu et al., 2023] 0.2em Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In Thirty-seventh Conference on Neural...

2023
[269]

Less: Selecting influential data for targeted instruction tuning

[Xia et al., 2024] 0.2em Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333, 2024

2024 arXiv
[270]

Efficient streaming language models with attention sinks

[Xiao et al., 2024] 0.2em Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In Proceedings of The Twelfth International Conference on Learning Representations, 2024

2024
[271]

Introduction to transformers: an nlp perspective

[Xiao and Zhu, 2023] 0.2em Tong Xiao and Jingbo Zhu. Introduction to transformers: an nlp perspective. arXiv preprint arXiv:2311.17633, 2023

2023 arXiv
[272]

Bagging and boosting statistical machine translation systems

[Xiao et al., 2013] 0.2em Tong Xiao, Jingbo Zhu, and Tongran Liu. Bagging and boosting statistical machine translation systems. Artificial Intelligence, 195: 0 496--527, 2013

2013
[273]

Sharing attention weights for fast transformer

[Xiao et al., 2019] 0.2em Tong Xiao, Yinqiao Li, Jingbo Zhu, Zhengtao Yu, and Tongran Liu. Sharing attention weights for fast transformer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19), pages 5292--5298, 2019

2019
[274]

An explanation of in-context learning as implicit bayesian inference

[Xie et al., 2022] 0.2em Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In Proceedings of International Conference on Learning Representations, 2022

2022
[275]

Deebert: Dynamic early exiting for accelerating bert inference

[Xin et al., 2020] 0.2em Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246--2251, 2020

2020
[276]

Wizardlm: Empowering large pre-trained language models to follow complex instructions

[Xu et al., 2024] 0.2em Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning R...

2024
[277]

[Yang et al., 2024] 0.2em An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

2024 arXiv
[278]

Xlnet: Generalized autoregressive pretraining for language understanding

[Yang et al., 2019] 0.2em Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019

2019
[279]

Tree of thoughts: Deliberate problem solving with large language models

[Yao et al., 2024] 0.2em Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024

2024
[280]

Unsupervised word sense disambiguation rivaling supervised methods

[Yarowsky, 1995] 0.2em David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd annual meeting of the association for computational linguistics, pages 189--196, 1995

1995
[281]

Towards better chain-of-thought prompting strategies: A survey

[Yu et al., 2023] 0.2em Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. Towards better chain-of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959, 2023

2023 arXiv
[282]

Alberti, S

[Zaheer et al., 2020] 0.2em Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, C. Alberti, S. Onta \ n \'o n, Philip Pham, Anirudh Ravula, Qifan Wang, L. Yang, and A. Ahmed. Big bird: Transformers for longer sequences. Advances in neural information processin...

2020
[283]

Swag: A large-scale adversarial dataset for grounded commonsense inference

[Zellers et al., 2018] 0.2em Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93--104, 2018

2018
[284]

Root mean square layer normalization

[Zhang and Sennrich, 2019] 0.2em Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

2019
[285]

Igniting language intelligence: The hitchhiker's guide from chain-of-thought reasoning to language agents

[Zhang et al., 2023] 0.2em Zhuosheng Zhang, Yao Yao, Aston Zhang, Xiangru Tang, Xinbei Ma, Zhiwei He, Yiming Wang, Mark Gerstein, Rui Wang, Gongshen Liu, and Hai Zhao. Igniting language intelligence: The hitchhiker's guide from chain-of-thought reasoning to language agents. ar...

2023 arXiv
[286]

Automatic chain of thought prompting in large language models

[Zhang et al., 2023] 0.2em Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, 2023 b

2023
[287]

Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning

[Zhao et al., 2024] 0.2em Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. arXiv preprint arXiv:2402.04833, 2024

2024 arXiv
[288]

Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen

[Zhao et al., 2023] 0.2em Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyu...

2023 arXiv
[289]

Lima: Less is more for alignment

[Zhou et al., 2023] 0.2em Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023 a

2023 arXiv
[290]

Le, and Ed H

[Zhou et al., 2023] 0.2em Denny Zhou, Nathanael Sch \" a rli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In Proceedings of ...

2023
[291]

Bert loses patience: Fast and robust inference with early exit

[Zhou et al., 2020] 0.2em Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33: 0 18330--18341, 2020

2020
[292]

Large language models are human-level prompt engineers

[Zhou et al., 2023] 0.2em Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2023 c

2023
[293]

Neural architecture search with reinforcement learning

[Zoph and Le, 2016] 0.2em Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In Proceedings of International Conference on Learning Representations, 2016

2016
[294]

Rethinking pre-training and self-training

[Zoph et al., 2020] 0.2em Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. Advances in neural information processing systems, 33: 0 3833--3845, 2020

2020
[295]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[296]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

write newline shortlabel

" write newline shortlabel "] 0.2em " * write "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format....

[2] [2]

Etc: Encoding long and structured inputs in transformers

[Ainslie et al., 2020] 0.2em Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 268--284, 2020

2020

[3] [3]

Gqa: Training generalized multi-query transformer models from multi-head checkpoints

[Ainslie et al., 2023] 0.2em Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895--4901, 2023

2023

[4] [4]

u rek et al., 2023] 0.2em Ekin Aky \

[Aky \"u rek et al., 2023] 0.2em Ekin Aky \"u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In Proceedings of The Eleventh International Conference on Learning Representations, 2023

2023

[5] [5]

Revisiting neural scaling laws in language and vision

[Alabdulmohsin et al., 2022] 0.2em Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35: 0 22300--22312, 2022

2022

[6] [6]

cosmopedia: how to create large-scale synthetic data for pre-training

[Allal et al., 2024] 0.2em Loubna Ben Allal, Anton Lozhkov, and Daniel van Strien. cosmopedia: how to create large-scale synthetic data for pre-training. https://huggingface.co/blog/cosmopedia, 2024

2024

[7] [7]

The falcon series of open language models

[Almazrouei et al., 2023] 0.2em Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \' e rouane Debbah, \' E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models. arXiv preprint a...

arXiv 2023

[8] [8]

Neural module networks

[Andreas et al., 2016] 0.2em Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39--48, 2016

2016