A Survey on Large Language Models with some Insights on their Capabilities and Limitations

Andrea Matarazzo; Riccardo Torlone

REVIEW 4 major objections 7 minor 11 cited by

A Survey on Large Language Models with some Insights on their Capabilities and Limitations

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that code in pre-training data helps elicit chain-of-thought and program-of-thought reasoning in LLMs, and reports experiments on Llama-family models using grade-school math benchmarks.

desk verdict A broad but error-prone LLM survey whose only novel claim (code data aids CoT emergence) sits in a missing Section 5, so the paper as supplied is a restatement with no verifiable new result. read the letter →

arxiv 2501.04040 v2 pith:YDZXDL2C submitted 2025-01-03 cs.CL cs.AIcs.LGcs.NE

Andrea Matarazzo , Riccardo Torlone This is my paper · ORCID

classification cs.CLcs.AIcs.LGcs.NE

keywords largelanguagemodelsLlamachain-of-thoughtpromptingprogram-of-thoughtpre-trainingdatacompositioncodeinemergentabilitiesGSM8k

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is mostly a survey of large language models, but it also advances a specific empirical claim: the presence of code in pre-training data helps elicit Chain-of-Thought (writing intermediate reasoning steps) and Program-of-Thought (writing code to compute the answer) reasoning. On Llama-family models, tested on grade-school arithmetic word problems from GSM8k and a harder variant called gsm-hard, the authors report that models with more code exposure perform better when prompted to reason step by step or to write a program that computes the answer. If this holds, data composition is a direct lever on reasoning ability, not just a correlate of model scale. The survey around the claim maps how LLMs are built, adapted, and applied, which is the context that makes the data-composition question worth settling.

What carries the argument

The load-bearing object is the pre-training data mixture, and within it the code fraction, treated as an independent variable that can be read off the documented training mixtures of Llama-family models. Chain-of-Thought prompting—asking the model to write its intermediate reasoning steps—and Program-of-Thought prompting—asking it to write a program whose execution yields the answer—are the probes that reveal whether the code in pre-training has left a trace. GSM8k and gsm-hard supply the grade-school math problems on which the probes are scored. The argument moves by comparing models whose corpora contain different reported code proportions—about 4.5% for the original Llama mix, roughly 8% for Llama 2, and about 17% for Llama 3—and checking whether step-by-step accuracy tracks that proportion.

What would settle it

A controlled pre-training study that keeps architecture, tokenizer, compute, and non-code data identical, varies only the code fraction—for example 0%, 5%, and 15%—and finds no monotonic improvement in Chain-of-Thought or Program-of-Thought accuracy on GSM8k and gsm-hard would falsify the paper's central claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that step-by-step reasoning in large language models is not just a matter of scale or prompting: it is shaped by what was in the pre-training corpus, with code singled out as the ingredient that makes Chain-of-Thought and Program-of-Thought abilities emerge or become easier to elicit. The supporting experiments use the public Llama family, whose documented training mixtures contain different amounts of code, and probe them on GSM8k and gsm-hard. Under Chain-of-Thought prompting the model is asked to produce intermediate natural-language reasoning steps; under Program-of-Thought prompting it is asked to write an executable program. The authors interpret the observed performance ordering as evidence that code-heavy pre-training is an enabling factor for these reasoning behaviors.

Load-bearing premise

The argument depends on attributing the observed GSM8k and gsm-hard performance differences to the code proportion in pre-training data, while model size, alignment tuning, tokenizer, and benchmark familiarity are not controlled across the compared models.

Editorial extensions

If this is right

Data curation becomes a direct lever on reasoning ability: increasing code in the corpus could improve Chain-of-Thought and Program-of-Thought performance without increasing model size.
General-purpose corpora may need to include code even when the intended applications are not programming, because code may teach the structured, compositional thinking that step-by-step prompting relies on.
The hypothesis is checkable on other openly documented model families: their reported code fractions should predict their relative GSM8k and gsm-hard performance under Chain-of-Thought and Program-of-Thought prompting.
The two probing styles matter: models with more code exposure may show their advantage most clearly when reasoning is delegated to executable programs rather than expressed in natural language.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A decisive test would be a controlled pre-training run that fixes architecture, tokenizer, compute, and non-code data, varying only the code fraction; the paper's between-model comparison cannot fully separate code from model size and alignment.
If the causal story holds, adding code to a training mix could give smaller models part of the reasoning gain currently attributed to scale, which would make step-by-step behavior cheaper to obtain.
The same reasoning suggests a caution for evaluation: gains on GSM8k and gsm-hard should be checked against code-form or solution leakage in the corpus before being credited to reasoning ability.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a broad survey of LLMs that advertises one original empirical contribution (Section 5: how code in pretraining data influences the emergence of chain-of-thought and program-of-thought reasoning). That section is not in the supplied manuscript, so the paper as reviewed is a restatement of known material with no verifiable new result. If the actual experiments are as described—comparing off-the-shelf Llama variants in LMStudio—the claim would not survive scrutiny, because Llama 1, 2, 3, and CodeLlama differ in size, training tokens, architecture, tokenizer, alignment, and code share all at once.

What the paper does well: it is a competent compilation. The survey covers model families, scaling laws, pre-training data, tokenization, instruction tuning, RAG, CoT, and planning in a logical order, and it cites most of the relevant literature. Someone new to LLMs could use it as a map. The idea that code in the training mix might enable step-by-step reasoning is a real research question, and the authors frame it clearly.

The soft spots are not subtle. First, the visible text contains concrete errors: Table 2 lists 'LLaMA 3.1 504B' (no such model; the largest is 405B), the header 'Casual decoder' appears in Section 3.5, and the BloombergGPT tokenizer description ends with 'only 2 17 tokens' (missing an exponent). These are minor individually but suggest careless copyediting. Second, the paper contradicts itself on whether model size is decisive for CoT: Section 2.1 leans on the >60B threshold, Section 4.2.2 says size is not decisive, and the empirical section never reconciles this. Third, and most important, the central empirical claim is unverifiable as supplied. Even the described protocol—GSM8k/gsm-hard accuracy on off-the-shelf Llama checkpoints—cannot separate code proportion from scale, alignment, and benchmark contamination. The 'Plan of Thought' acronym is also wrong in the abstract: the literature calls it Program of Thought, which makes me wonder how carefully Section 5 was written.

Who is this for? A student wanting a broad orientation, not a researcher needing accurate details. The survey portion could be made useful, but as it stands it does not deserve a serious referee without the actual experiments and a round of corrections. If the authors include Section 5 and tighten the controls, I would look again.

My recommendation: treat this as a desk reject with an invitation to resubmit a corrected version that includes the empirical details. It is not a paper that should be sent to referees in its current form.

Referee Report

4 major / 7 minor

Summary. The paper is a broad survey of large language models: it reviews the historical development of LMs, scaling laws, prominent model families (BERT, T5, GPT, Llama, Gemma, Claude), domain-specific applications, pre-training data and preprocessing, adaptation via instruction and alignment tuning, transformer architectures, and usage techniques such as in-context learning, chain-of-thought prompting, program-of-thoughts, planning, and retrieval-augmented generation. The abstract and Section 1 present the paper's advertised original contribution as an empirical investigation of whether the proportion of code in pre-training data influences the emergence of Chain-of-Thought and Program-of-Thought abilities, with experiments on Llama-family models evaluated on GSM8k and gsm-hard. In the version of the manuscript made available for review, Sections 1.2, 1.3, and the table of contents describe this study, but the body text of Section 5 itself is not present; the provided text ends midway through Section 3.5.4. The survey portions collate many well-known results and cite numerous primary sources, but they also contain several concrete errors, including an incorrect model size in Table 2 and garbled text in the tokenizer description.

Significance. If the empirical claim about code in pre-training data being a causal or enabling factor for CoT/PoT emergence were established with adequate controls, it would be a useful contribution to the ongoing debate on what drives apparent emergent reasoning in LLMs, and it would complement earlier observations about code training and step-by-step reasoning. The survey parts of the paper could serve as a broad introductory reference, especially for readers wanting a single narrative covering scaling laws, model families, adaptation, and prompting strategies. The paper's strengths are its breadth, its many pointers to primary sources, and its generally clear organization; its weaknesses are the absence of the Section 5 evidence from the reviewed text, the confounding factors inherent in the described Llama-variant comparison, and a number of factual and typographical errors that currently reduce the reliability of the survey as a reference. There are no machine-checked proofs or reproducible code artifacts in the visible text, so the paper must be judged on the quality of its synthesis and on the credibility of its empirical section.

major comments (4)

[Abstract; Sections 1.2, 1.3, 5] The paper's central original claim—that the presence of code in pre-training data influences the emergence of CoT and PoT abilities—is announced in the abstract and in Section 1.3, but the actual content of Section 5 (subsections 5.1–5.4) is not present in the manuscript text supplied for review. Only the table of contents entries and the introductory description in Section 1.3 are visible. As reviewed, the claim is therefore unsupported: the reader cannot check the experimental setup, the prompts, the result tables, or the quoted generated text. If the full Section 5 exists in the authors' version, it must be included in the review copy; if it does not, the manuscript currently promises an empirical contribution it does not deliver.
[Section 1.3; Section 5.2; Tables 2 and 4] The experimental design described in the paper cannot isolate the proportion of code in pre-training data as the cause of CoT/PoT performance differences. Comparing off-the-shelf Llama variants, as described in Section 1.3, means that model size, training-token count, architecture details, tokenizer, context length, and alignment status all vary together with code proportion: Table 4 reports only 4.5% code for Llama 1, while the text reports roughly 8% for Llama 2, 17% for Llama 3, and about 85% code for CodeLlama, but these models also differ in parameters (7B–70B), training tokens (1T–15T), and other hyperparameters visible in Table 2. Without matched-size models, or explicit controls for scale and training budget, any observed accuracy differences on GSM8k or gsm-hard cannot be attributed to code exposure. The authors need to either provide such controls or substantially weaken the causal claim to a correlational one.
[Section 2.1 vs. Section 4.2.2] The manuscript is internally inconsistent about the role of model size in CoT ability, and this inconsistency is load-bearing for the Section 5 comparison. Section 2.1 reports that CoT prompting yields gains mainly when model size exceeds 60B and that advantages become more pronounced above 100B, while Section 4.2.2 is described as saying that size is not a deciding factor. Since the Llama-family comparison described in Section 1.3 crosses exactly the size range where this debate matters (7B–70B), the empirical section must explicitly address how scale is controlled or otherwise accounted for. As it stands, the reader cannot tell whether a performance gap is due to code proportion, model size, or both.
[Section 5.2; GSM8k/gsm-hard] Using accuracy on GSM8k and gsm-hard as the measure of reasoning ability requires a contamination analysis, because these datasets are public, widely reproduced, and frequently included in web scrapes used for pre-training. The paper's own Section 3.3.2 discusses the importance of deduplication and contamination, but the described Section 5 experiments do not mention any decontamination or holdout procedure. Higher scores on these benchmarks could in part reflect memorization rather than the emergence of step-by-step reasoning. The authors should report overlap analysis between the evaluation items and the pre-training corpora of the tested Llama variants, or use a fresh or adversarially filtered evaluation split.

minor comments (7)

[Table 2] The table lists 'LLaMA 3.1 504B'; the correct size is 405B. The same table also appears to conflate Llama 3 and Llama 3.1 training-token counts; the caption should cite the specific source for each row.
[Section 3.5.2 and Table 21] The heading 'Casual decoder' should read 'Causal decoder'; the same misspelling appears in the surrounding text and in Table 21's model-card entries. This is a terminology error in a central architectural taxonomy.
[Section 2.4.2 (BloombergGPT)] The tokenizer description reads 'The result was cut from a tokenizer with 7 million tokens to only 2 17 tokens', which is garbled; it should presumably read '2^17 tokens' or a specific vocabulary size. Please correct the typesetting and state the final vocabulary size explicitly.
[Section 2.3.3] There is an obvious typo 'GTP-4 GPT-4'; also 'GTP-3' appears earlier in the same section. Please run a spelling pass over model names.
[Section 2.3.5 and elsewhere] The model name 'Minstral' should be 'Mistral', and 'seetings' should be 'settings'. Several other typos appear throughout; the paper would benefit from professional copyediting.
[Section 2.4.2] The text says 'Within the four FinLLMs in Figure 8', but the figure containing BloombergGPT, FinMA, InvestLM, and FinGPT is Table 8, not Figure 8.
[Abstract and Section 5.2] The abstract defines 'PoT' as 'Plan of Thought', while Section 5.2 uses 'Program of Thought'. These should be reconciled, and the standard term 'Program of Thoughts' should be used consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a survey compiling external results, and the only original empirical claim (Section 5) is absent from the manuscript; it is at most an uncontrolled correlation, not a derivation from its own inputs.

full rationale

This manuscript is primarily a survey: sections 2-4 summarize external scaling laws, architectures, benchmarks, and prompting techniques, and cite the original sources (Kaplan et al., Hoffmann et al., Wei et al., Touvron et al., etc.) rather than deriving new results. The only original claim announced in the abstract and Section 1.3 is the Section 5 hypothesis that code in pretraining data contributes to CoT/PoT emergence, evaluated on Llama-family models with GSM8k and gsm-hard. The body of Section 5 is not present in the supplied text, so there is no equation, fitted parameter, or benchmark split that could be shown to reduce to an input. The described design—comparing off-the-shelf Llama versions with different reported code shares—is a confounded observational comparison, but confounding is a validity problem, not a circularity problem: code share is taken from released model documentation, the outcome is measured accuracy on reasoning benchmarks, and neither is defined in terms of the other. No load-bearing argument rests on a self-citation: the authors do not cite their own prior work, and the cited scaling laws and CoT results are external and independently checkable. Consequently, there is no self-definitional step, no fitted-input-called-prediction, and no imported uniqueness theorem; the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

No new free parameters or invented entities; the only axiom is faithful reporting of cited sources.

assumptions (1)

domain assumption Cited primary sources are represented accurately.
The survey's value depends on faithful reporting of scaling laws, model cards, and benchmark numbers; visible slips undermine this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Survey on Large Language Models with some Insights on their Capabilities and Limitations." pith.science (2026). https://pith.science/paper/YDZXDL2C

@misc{pith2026250104040,
  author       = {Pith},
  title        = {Pith review of: A Survey on Large Language Models with some Insights on their Capabilities and Limitations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YDZXDL2C}},
  note         = {Machine review of arXiv:2501.04040}
}

read the original abstract

The rapid advancement of artificial intelligence, particularly with the development of Large Language Models (LLMs) built on the transformer architecture, has redefined the capabilities of natural language processing. These models now exhibit remarkable performance across various language-related tasks, such as text generation, question answering, translation, and summarization, often rivaling human-like comprehension. More intriguingly, LLMs have demonstrated emergent abilities extending beyond their core functions, showing proficiency in tasks like commonsense reasoning, code generation, and arithmetic. This survey paper explores the foundational components, scaling mechanisms, and architectural strategies that drive these capabilities. Emphasizing models like GPT and LLaMA, we analyze the impact of exponential data and computational growth on LLM performance, while also addressing the trade-offs associated with scaling. We also examine LLM applications across sectors, such as healthcare, finance, education, and law, highlighting their adaptability and potential to solve domain-specific challenges. Central to this work are the questions of how LLMs generalize across diverse tasks, exhibit planning, and reasoning abilities, and whether these emergent abilities can be systematically elicited or enhanced. In particular, we provide some insights into the CoT (Chain of Thought) and PoT (Plan of Thought) abilities within LLMs, focusing on how pre-training data influences their emergence. Additionally, we investigate LLM-modulo frameworks that integrate external systems, allowing LLMs to handle complex, dynamic tasks. By analyzing these factors, this paper aims to foster the ongoing discussion on the capabilities and limits of LLMs, promoting their responsible development and application in novel and increasingly complex environments.

Figures

Figures reproduced from arXiv: 2501.04040 by the authors.

**Figure 1.** Two examples of in-context learning, where a language model (LM) is given a list of training examples (black) and a test input (green) and asked to make a prediction (orange) by predicting the next tokens/words to fill in the blank. Source: Lab [288]. few-shot learning 5 , and even one-shot6 learning7 . Three typical examples of emergent abilities are: (a) In-context learning: this ability has been formally observed… view at source ↗

**Figure 2.** Left: scaling law. Model performance increases linearly as the model size increases exponentially. Right: emergent abilities show a phase change at a certain scale where the performance suddenly increases. Source: Fu [267]. Despite propelling the field of LLMs to new heights, the scaling law comes with computational challenges. Training huge models requires significant computational resources, encompassing proces… view at source ↗

**Figure 3.** A diagram showing the evolution of publicly available LLMs. Source: Zhao et al. [364]. bidirectional Transformer encoder based on the original transformer architecture introduced by Vaswani et al. [334]. Unlike its predecessors, BERT analyses text in both directions (leftto-right and right-to-left), providing a more nuanced understanding of language context. This bi-directionality enables BERT to achieve state-of-t… view at source ↗

Figures from the paper (65 more)

**Figure 4.** Figure 4: BERT Architecture: The bottom layer contains the embedding representations E1, E2, . . . EN , which encode input tokens and serve as the input to the transformer layers (Trm). Each transformer bidirectionally processes the input embeddings, and the final output is used…

**Figure 5.** Figure 5: A diagram of the T5 text-to-text framework. Every task – including translation, question answering, and classification – is cast as feeding the model text as input and training it to generate some target text. This approach allows the same model, loss function, hyperpa…

**Figure 6.** Figure 6: o1 greatly improves over GPT-4o on challenging reasoning benchmarks. Solid bars show pass@1 accuracy and the shaded region shows the performance of majority vote (consensus) with 64 samples. Source: OpenAI [383]. o1 has demonstrated proficiency in various domains, incl…

**Figure 7.** Figure 7: The Code Llama 70B specialization pipeline. The different fine-tuning stages are annotated with the number of tokens seen during training. Infilling-capable models are marked with the ⇄ symbol. Source: Rozi`ere et al. [384]. While most of the code generation models are…

**Figure 8.** Figure 8: Comparison of Code Llama models versus an identical model trained from scratch. Source: Rozi`ere et al. [384]. 2. an attention mask that prevents self-attention between different documents within the same sequence. This change has limited impact during standard pre-tra…

**Figure 9.** Figure 9: Gemma models exhibit superior performance in language understanding and reasoning tasks compared to larger models. Source: Team et al. [385]. In comparative benchmarks, Gemma models have demonstrated capabilities that exceed those of larger parameter models, such as Ll…

**Figure 10.** Figure 10: Gemma models are designed to be lightweight and efficient, making them accessible to a wide range of developers and applications. Source: Banks and Warkentin [372]. ethical considerations, such as bias in language models, remain an area of concern and active developme…

**Figure 11.** Figure 11: Large Language Models (LLMs) have revolutionized healthcare by enhancing diagnostic accuracy, clinical decision support, and patient engagement. Source: Singhal et al. [213]. Here are some key applications of LLMs in the financial sector: 1. Algorithmic Trading: LLMs …

**Figure 12.** Figure 12: Timeline showing the evolution of selected PLM/LLM releases from the general domain to the financial domain. Source: Lee et al. [380]. Model Backbone Paras. PT Techniques PT Data Size Evaluation Task Dataset O.S. Model PT IFT BloombergGPT [350] BLOOM 50B PT, PE (G) 3…

**Figure 13.** Figure 13: Prompts used in Blair-Stanek, Holzenberger, and Durme [253] to pose SARA test cases to GPT-3. The top boxes, in orange, contain statutes (optional). Example cases are in blue; in zeroshot, no example cases exist. At the bottom, in green, are test cases. The text high…

**Figure 14.** Figure 14: Using only the minimal labelled data points available, a supervised model may learn a decision boundary that will generalize poorly and be prone to misclassifying new examples. Source: Bergmann [252]. One significant advantage of supervised pre-training is its potenti…

**Figure 15.** Figure 15: LEFT: original labelled and unlabeled data points. RIGHT: using label propagation, the unlabeled data points have been assigned pseudo-labels. Source: Bergmann [252]. Active learning is another inductive learning method that involves iteratively selecting the most inf…

**Figure 16.** Figure 16: Commonly-used data sources for training and evaluating Large Language Models (LLMs). Source: Zhao et al. [364]. • Conversation text: Conversation text, including chat logs and social media interactions, provides a rich source of informal language and colloquial expre…

**Figure 17.** Figure 17: Common data preprocessing steps for training Large Language Models (LLMs). Source: Zhao et al. [364]. 3.3.1 Quality Filtering. The first step in data preprocessing is quality filtering, where the data is cleaned to remove irrelevant or low-quality content. Existing w…

**Figure 18.** Figure 18: Overview of instruction tuning. Source: Zhao et al. [364]. least 62B, though a much smaller size might suffice for some specific tasks (e.g., MMLU). Instruction tuning has been widely applied also in other models like Instruct-GPT [205] and GPT-4 [316]. Other experime…

**Figure 19.** Figure 19: Top: overview of instruction tuning and FLAN. Instruction tuning finetunes a pretrained language model on a mixture of tasks phrased as instructions. Evaluation of unseen task type at inference time (i.e., evaluate the model on natural language inference (NLI) when n…

**Figure 20.** Figure 20: Overview of FLAN instruction tuning with and without exemplars (i.e., zero-shot and few-shots) and with and without CoT. Following evaluation on unseen tasks. Source: Chung et al. [156] [PITH_FULL_IMAGE:figures/full_fig_p056_20.png]

**Figure 21.** Figure 21: Three main approaches to construct instruction-formatted instances. Source: Zhao et al. [364]. models – most training instances come from public NLP datasets that often lack instructional diversity and do not align well with actual human needs. This approach helps to …

**Figure 22.** Figure 22: Some of the mainstream LLMs models by type. Mainstream architectures can be further categorized into three major types: encoderdecoder, casual decoder and prefix decoder, as shown in [PITH_FULL_IMAGE:figures/full_fig_p063_22.png]

**Figure 23.** Figure 23: A comparison of the attention patterns in three mainstream architectures. Here, the blue, green, yellow and grey rounded rectangles indicate the attention between prefix tokens, attention between prefix and target tokens, attention between target tokens, and masked at…

**Figure 24.** Figure 24: The full model architecture of the transformer. Source: Weng [57]. The position-wise FFN sub-layer is a two-layer feed-forward network with a ReLU activation function between the layers. Given a sequence of vectors h1, h2, . . . , hn, the computation of a position-wis…

**Figure 25.** Figure 25: Illustration of different LayerNorm structures in Transformers. Source: Ding et al. [109]. Sandwich-LN [109] is a configuration that combines the advantages of both pre-LN and post-LN by placing the normalization layer both before and after the residual blocks. This c…

**Figure 26.** Figure 26: (left) Scaled Dot-Product Attention. (right) Multi-head attention consists of several attention layers running in parallel. Source: Vaswani et al. [334]. scaled dot-product attention function used in Vaswani et al. [334] is defined as follows: 71 [PITH_FULL_IMAGE:fig…

**Figure 27.** Figure 27: PagedAttention: KV Cache is partitioned into blocks. Source: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention [335]. Paging increases the GPU memory utilization and enables efficient memory sharing in parallel sampling ( [PITH_FULL_IMAGE:figures/full_fig_…

**Figure 28.** Figure 28: PagedAttention: example of parallel sampling. Source: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention [335]. To put all these discussions together, Zhao et al. [364] summarize the suggestions from existing literature for detailed configuration. For stronge…

**Figure 29.** Figure 29: The Hyena operator is defined as a recurrence of two efficient subquadratic primitives: an implicit long convolution h (i.e., Hyena filters parameterized by a feed-forward network) and multiplicative element-wise gating of the (projected) input. The depth of the recu…

**Figure 30.** Figure 30: An illustration of four different parameter-efficient fine-tuning methods. MHA and FFN denote the multi-head attention and feed-forward networks in the Transformer layer, respectively. Source: Zhao et al. [364]. Adapter tuning Adapter tuning is a parameter-efficient t…

**Figure 31.** Figure 31: On the left, the architecture of the adapter module and its integration with the Transformer. The adapter module is added twice to each Transformer layer. On the right, the adapter module consists of a feed-forward network with a bottleneck layer and a residual conne…

**Figure 32.** Figure 32: Illustration of the prefix-tuning method, which freezes the Transformer parameters and only optimizes the prefix (the red prefix blocks). Consequently, it only needs to store the prefix for each task, making prefix-tuning modular and space-efficient. Note that each ve…

**Figure 33.** Figure 33: Illustration of the prompt tuning method, which only requires storing a small task-specific prompt for each task and enables mixed-task inference using the original pre-trained model. With model tuning, each copy of tuned models requires a copy of billions of paramete…

**Figure 34.** Figure 34: In-context learning contrasted with traditional fine-tuning. Source: Brown et al. [88] learning new tasks from k examples alone, without relying on task reformatting (e.g., reducing everything to question answering) or task-specific templates (e.g., converting differe…

**Figure 35.** Figure 35: Two examples of in-context learning, where a language model (LM) is given a list of training examples (black) and a test input (green) and asked to make a prediction (orange) by predicting the next tokens/words to fill in the blank. Source: Lab [288] In-context Learni…

**Figure 36.** Figure 36: Taxonomy of in-context learning. The training and the inference stage are two main stages for ICL. During the training stage, existing ICL studies mainly take a pre-trained LLM as the backbone and optionally warm up the model to strengthen and generalize the ICL abili…

**Figure 37.** Figure 37: Chain-of-Thought reasoning for GSM8k math word problem. The prompt is coloured black, and the reasoning path produced by the language model is coloured teal. This reasoning path contains two reasoning steps. Source: Li et al. [295] of prompts with a series of intermed…

**Figure 38.** Figure 38: A comparative illustration of in-context learning (ICL) and chain-of-thought (CoT) prompting. ICL prompts LLMs with a natural language description, several demonstrations, and a test query, while CoT prompting involves a series of intermediate reasoning steps in promp…

**Figure 39.** Figure 39: The DIVERSE approach for CoT. Source: Li et al. [295] enhance the performance of LLMs on complex reasoning tasks. The proposed method, DIVERSE82, generates diverse CoTs by leveraging a self-ensemble approach that alternates between selection and inference. It has th…

**Figure 40.** Figure 40: A: Chain of thoughts (in blue) are intermediate reasoning steps towards a final answer. The input of CoT prompting is a stack of a few (often 8) CoT cases before a test question. Then, the language model will continue generating an output CoT for the test question. B:…

**Figure 41.** Figure 41: Complexity-based Consistency for CoT. During decoding, it samples N reasoning chains from the language model (N = 5 here) and takes the majority answer over the K (K = 3 here) most complex generated chains. Source: Fu et al. [163] [PITH_FULL_IMAGE:figures/full_fig_p0…

**Figure 42.** Figure 42: Zero-Shot-CoT [285] (using the “Let’s think step by step” prompt) and Manual-CoT[230] (using manually designed demonstrations one by one) with example inputs and outputs of an LLM. Source: Zhang et al. [243] the prompt, eliminating the need for annotated CoT datasets …

**Figure 43.** Figure 43: demonstrations (on the right) are automatically constructed one by one (total: k) using an LLM with the “Let’s think step by step” prompt. Source: Zhang et al. [243] the representative selection algorithm can select the most representative questions from each cluster …

**Figure 44.** Figure 44: Program-of-Thoughts (PoT) for solving math word problems. The input is a math word problem, and the output is a program that can solve the problem. Source: Chen et al. [259] The origins of CoT ability are widely hypothesized to be elicited by training on code since th…

**Figure 45.** Figure 45: The general framework of prompt-based planning. Source: Zhao et al. [364] In this paradigm, there are three main components: the planner, the executor, and the environment97. The first component is the planner, which generates a plan of action to solve the task. The p…

**Figure 46.** Figure 46: Example inputs and outputs of GPT-3 with (a) Zero-shot-CoT prompting, (b) Plan-andSolve (PS) prompting, and (c) answer extraction prompting. While Zero-shot-CoT encourages LLMs to generate multi-step reasoning with “Let’s think step by step”, it may still generate wr…

**Figure 47.** Figure 47: Least-to-most prompting teaches language models how to solve a complex problem by decomposing it to a series of simpler subproblems. It consists of two sequential stages: (1) decomposition and (2) sequentially solving subproblems. The answer to the second subproblem …

**Figure 48.** Figure 48: The DECOMP framework. Source: Khot et al. [175] In DECOMP, the core is a decomposer LLM that tries to solve a complex task by generating a prompting program P. Each step of P directs a simpler sub-query to a function in an auxiliary set of sub-task functions F availab…

**Figure 49.** Figure 49: On the left: Exact Match results on the k-th letter concatenation task (k=3) using space as a delimiter with different numbers of words in the input. On the right: Exact Match results on reversing sequences. Incorporating CoT in DECOMP greatly increases the ability of…

**Figure 50.** Figure 50: Example prompt for the mathematical reasoning tasks from the GSM8k benchmark. Source: Gao et al. [164] mathematical word problems and symbolic reasoning tasks [164] as shown in [PITH_FULL_IMAGE:figures/full_fig_p105_50.png]

**Figure 51.** Figure 51: An example for a PaL prompt in the Colored Objects task. Source: Gao et al. [164] – a version of GSM8k contains larger numbers (i.e., up to 7 digits). Other interesting results come from symbolic reasoning tasks from BIG-Bench Hard: the Colored Objects103 and the Peng…

**Figure 52.** Figure 52: Self-planning generation phases (i.e., planning and implementation phases). Source: Jiang et al. [377] In the second stage, the implementation phase, the plan generated in the first stage guides the code generation. The plan y is concatenated with intent x and fed int…

**Figure 53.** Figure 53: (1) Comparison of 4 prompting methods, (a) Standard, (b) Chain-of-thought (CoT, Reason Only), (c) Act-only, and (d) ReAct (Reason+Act), solving a HotpotQA [60] question. Source: Chen et al. [260] [PITH_FULL_IMAGE:figures/full_fig_p109_53.png]

**Figure 54.** Figure 54: ChatCoT strategy illustrated to solve a mathematical problem. The conversational knowledge memory is initialized to provide tools, task and reasoning format knowledge. Then, the toolaugmented reasoning step is iterated multiple times to perform step-by-step reasonin…

**Figure 55.** Figure 55: Diagram demonstrating various problem-solving methodologies using LLMs. Each rectangle represents a distinct thought, forming an integral step towards resolving a problem. Source: Yao et al. [359] Wiring107, and Crosswords108. These tasks necessitate a blend of deduc…

**Figure 56.** Figure 56: Overview of the DEPS interactive plannet architecture. Source: Wang et al. [344] as an explainer to locate the errors in the previous plan. Finally, a planner will refine the plan using the descriptor and explainer information. To improve the feasibility of generated …

**Figure 57.** Figure 57: Reflexion works on decision-making, programming, and reasoning tasks. Source: Shinn et al. [322] [PITH_FULL_IMAGE:figures/full_fig_p112_57.png]

**Figure 58.** Figure 58: (a) Diagram of Reflexion. (b) Reflexion reinforcement algorithm. Source: Shinn et al. [322] process are the notion of short-term and long-term memory. At inference time, the Actor conditions its decisions on short and long-term memory, similar to how humans remember f…

**Figure 59.** Figure 59: Generative agent architecture. Agents perceive their environment, and all perceptions are saved in a comprehensive record of the agent’s experiences called the memory stream. Based on their perceptions, the architecture retrieves relevant memories and uses those retri…

**Figure 60.** Figure 60: Adding and retrieving skills from the skill library in Voyager. Source: Sun et al. [324] [PITH_FULL_IMAGE:figures/full_fig_p114_60.png]

**Figure 61.** Figure 61: Overview of MemoryBank. The memory storage stores past conversations, summarized events and user portraits, while the memory updating mechanism updates the memory storage. Memory retrieval recalls relevant memory. Source: Zhong et al. [367] Bank [367] incorporates a m…

**Figure 62.** Figure 62: LLMs serve as idea generators, while various external critics, each specializing in different aspects, evaluate and provide feedback on the proposed plan. Source: Kambhampati et al. [379] hampati et al. [379] argue that auto-regressive large language models (LLMs) lac…

**Figure 63.** Figure 63: LLM Modulo Framework adapted for Travel Planning. Source: Kambhampati et al. [379] LLM-Modulo framework to this benchmark by operationalizing hard constraints (e.g., budget constraints) or commonsense constraints (e.g., suggesting diverse attractions to visit) as crit…

**Figure 64.** Figure 64: Final Pass rates of models across LLM Modulo Iterations. Source: Kambhampati et al. [379] 3. Domain Adaptability: RAG enables LLMs to integrate domain-specific information, improving performance in specialized areas like law, medicine, and engineering. RAG systems are…

**Figure 65.** Figure 65: Technology tree of RAG research. The stages of involving RAG mainly include pretraining, fine-tuning, and inference. With the emergence of LLMs, research on RAG initially focused on leveraging the powerful in context learning abilities of LLMs, primarily concentratin…

**Figure 66.** Figure 66: Retrieval-Augmented Generation (RAG) Framework mainly consists of 3 steps. 1) Indexing. Documents are split into chunks, encoded into vectors, and stored in a vector database. 2) Retrieval. Retrieve the Top k chunks most relevant to the question based on semantic sim…

**Figure 67.** Figure 67: Example of a gsm-hard problem. The reasoning steps are expressed as code. which require highly symbolic reasoning skills. The previous paragraph explored a similar approach (see Par. 4.4.3). An example of a gsm-hard problem, reasoning steps and solution is shown in th…

**Figure 68.** Figure 68: The prompt format with role-based messages. In the 5-shot prompting, the model is given five examples of the task, and the structure is similar to what we see in [PITH_FULL_IMAGE:figures/full_fig_p128_68.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Generalizable and Computational Efficient Channel Extrapolation for 6G: A Configurable AI-Driven Framework Built from a Modular Perspective
eess.SP 2026-08 conditional novelty 6.0 of 10

A three-stage modular AI framework, pretrain, cluster experts, and learn routing, improves channel extrapolation accuracy and cuts FLOPs in simulated 6G scenarios.
EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detection
cs.CL 2025-05 conditional novelty 6.0 of 10

A new benchmark of 2,833 evasive text samples and 13,961 images shows current LLMs and VLMs frequently miss veiled policy violations in Chinese e-commerce ads.
On Technique Identification and Threat-Actor Attribution using LLMs and Embedding Models
cs.CR 2025-05 conditional novelty 6.0 of 10

GPT-4's TTP lists match human MITRE labels only 39% by Jaccard similarity, yet an embedding-based attribution model ranks the correct threat actor 7.55 on average out of 29, beating the random baseline of 15.
Large Language Models for Next-Generation Wireless Network Management: A Survey and Tutorial
cs.NI 2025-09 conditional novelty 4.0 of 10

A survey and tutorial that organizes LLM-enabled wireless network optimization into formulation, solution, and verification stages, with case studies drawn from the authors' own prior papers.
InsightBuild: LLM-Powered Causal Reasoning in Smart Building Systems
cs.LG 2025-07 reject novelty 4.0 of 10

InsightBuild uses Granger causality and structural pruning to rank sensor causes, then a fine-tuned LLaMA 2 model turns the ranked causes into text explanations that outperform baseline systems on annotated building e...
ALAS: A Stateful Multi-LLM Agent Framework for Disruption-Aware Planning
cs.AI 2025-05 reject novelty 4.0 of 10

ALAS combines role-specialized LLM agents, persistent state, and a local compensation protocol to produce disruption-tolerant schedules, reporting a 0.86% mean gap on a subset of Taillard instances and 19.09% on Demirkol-DMU.
Survey of GenAI for Automotive Software Development: From Requirements to Executable Code
cs.SE 2025-07 conditional novelty 3.0 of 10

A review of roughly 60 papers and 9 industry respondents finds GPT-family models dominate automotive code generation while requirements handling lags due to confidentiality constraints.
RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models
cs.CL 2025-06 reject novelty 3.0 of 10

RACE-Align generates preference pairs from RAG-grounded chain-of-thought answers and applies DPO to align a 1.7B model, showing improved reasoning scores in TCM QA but lacking statistical support.
From Transformers to Large Language Models: A systematic review of AI applications in the energy sector towards Agentic Digital Twins
cs.LG 2025-06 conditional novelty 3.0 of 10

A systematic review of 99 papers on transformers and LLMs in energy, ending with a proposal for LLM-powered Agentic Digital Twins for smart grids.
A Comprehensive Survey on Integrating Large Language Models with Knowledge-Based Methods
cs.CL 2025-01 conditional novelty 3.0 of 10

A narrative review of LLM knowledge integration that categorizes techniques and compiles benchmarks, but lacks a systematic method and contains unreliable citations.
A Survey of the State-of-the-Art in Conversational Question Answering Systems
cs.CL 2025-09 conditional novelty 2.0 of 10

A review that categorizes ConvQA components, techniques, models, and datasets, with no new experimental result.

Reference graph

Works this paper leans on

293 extracted references · 7 canonical work pages · cited by 11 Pith papers

[1]

More is Different: Broken Symmetry and the Nature of the Hier- archical Structure of Science

Philip W. Anderson. “More is Different: Broken Symmetry and the Nature of the Hier- archical Structure of Science”. In: (1972). url: http://www.lanais.famaf.unc.edu. ar/cursos/em/Anderson-MoreDifferent-1972.pdf

1972
[2]

Mitchell

Tom M. Mitchell. Machine Learning. McGraw-Hill, 1997

1997
[3]

Statistical Learning Theory

Vladimir Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998

1998
[4]

Transductive inference for text classification using support vector machines

Thorsten Joachims. “Transductive inference for text classification using support vector machines”. In: ICML. Citeseer. 1999

1999
[5]

Conditional random fields: Probabilistic models for segmenting and labeling sequence data

John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. “Conditional random fields: Probabilistic models for segmenting and labeling sequence data”. In: Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001) . Ed. by Carla E. Brodley and Andrea P. Danyluk. Morgan Kaufmann, 2001, pp. 282–289

2001
[6]

A Neural Probabilistic Language Model

Yoshua Bengio et al. “A Neural Probabilistic Language Model”. In: Journal of Machine Learning Research 3 (2003), pp. 1137–1155

2003
[7]

V AL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL

R. Howey, D. Long, and M. Fox. “V AL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL”. In: 16th IEEE International Conference on Tools with Artificial Intelligence (2004), pp. 294–301

2004
[8]

Conceptnet–a practical commonsense reasoning tool-kit

Hugo Liu and Push Singh. “Conceptnet–a practical commonsense reasoning tool-kit”. In: BT technology journal 22 (2004), pp. 211–226

2004

Show all 293 references

[9]

Learning with unlabeled data and its application to image re- trieval

Dengyong Zhou et al. “Learning with unlabeled data and its application to image re- trieval”. In: Proceedings of the 2004 ACM SIGKDD international conference on Knowl- edge discovery and data mining . ACM. 2004

2004
[10]

Semi-supervised Learning Literature Survey

Xiaojin Zhu. Semi-supervised Learning Literature Survey. University of Wisconsin-Madison Department of Computer Sciences, 2005

2005
[11]

Manifold regularization: A ge- ometric framework for learning from labeled and unlabeled examples

Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. “Manifold regularization: A ge- ometric framework for learning from labeled and unlabeled examples”. In: Journal of machine learning research. MIT Press. 2006

2006
[12]

One-Shot Learning of Object Categories

Li Fei-Fei, Rob Fergus, and Pietro Perona. “One-Shot Learning of Object Categories”. In: Proceedings of the 2006 Conference on Object Recognition (2006). url: http : / / vision.stanford.edu/documents/Fei-FeiFergusPerona2006.pdf. 152

2006
[13]

A tutorial on planning graph based reachability heuris- tics

D. Bryce and S. Kambhampati. “A tutorial on planning graph based reachability heuris- tics”. In: AI Mag. 28.1 (2007), pp. 47–83

2007
[14]

Semi-supervised Learning

Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised Learning. MIT Press, 2009

2009
[15]

The Path to Personalized Medicine

M. A. Hamburg and F. S. Collins. “The Path to Personalized Medicine”. In: New England Journal of Medicine 363.4 (2010), pp. 301–304. doi: 10 . 1056 / NEJMp1006304. url: https://sci-hub.se/10.1056/NEJMp1006304

2010 doi
[16]

Rectified Linear Units Improve Restricted Boltz- mann Machines

Vinod Nair and Geoffrey E. Hinton. “Rectified Linear Units Improve Restricted Boltz- mann Machines”. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10). 2010, pp. 807–814

2010
[17]

Deep Sparse Rectifier Neural Net- works

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. “Deep Sparse Rectifier Neural Net- works”. In: Proceedings of the Fourteenth International Conference on Artificial Intelli- gence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011 (2011)

2011
[18]

Thinking, Fast and Slow

Daniel Kahneman. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011

2011
[19]

Pseudo-Label: The Simple and Efficient Semi-supervised Learning Method for Deep Neural Networks

Dong-Hyun Lee. “Pseudo-Label: The Simple and Efficient Semi-supervised Learning Method for Deep Neural Networks”. In: ICML 2013 Workshop: Challenges in Represen- tation Learning (WREPL) (2013)

2013
[20]

Rectifier nonlinearities improve neural network acoustic models

Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. “Rectifier nonlinearities improve neural network acoustic models”. In: CoRR abs/1312.6026 (2013). arXiv: 1312.6026 [cs.LG]

2013 arXiv
[21]

Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov et al. “Distributed Representations of Words and Phrases and Their Compositionality”. In: Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a Meeting Held December 5-8, 2013,...

2013
[22]

Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov et al. “Efficient Estimation of Word Representations in Vector Space”. In: 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2013

2013
[23]

Neural machine translation by jointly learning to align and translate

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. “Neural machine translation by jointly learning to align and translate”. In:CoRR abs/1409.0473 (2014). arXiv: 1409. 0473 [cs.CL]

2014 arXiv
[24]

Good debt or bad debt: Detecting semantic orientations in economic texts

Pekka Malo et al. “Good debt or bad debt: Detecting semantic orientations in economic texts”. In: JASIST 65.4 (2014), pp. 782–796

2014
[25]

Domain adaption of named entity recognition to support credit risk assessment

Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. “Domain adaption of named entity recognition to support credit risk assessment”. In: Proceedings of ALTA Workshop. 2015, pp. 84–90

2015
[26]

Batch normalization: Accelerating deep network training by reducing internal covariate shift

Sergey Ioffe and Christian Szegedy. “Batch normalization: Accelerating deep network training by reducing internal covariate shift”. In: CoRR abs/1502.03167 (2015). arXiv: 1502.03167 [cs.LG]

2015 arXiv
[27]

Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

Yukun Zhu et al. “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books”. In: 2015 IEEE International Conference on Computer Vision (ICCV) . IEEE Computer Society. Santiago, Chile, 2015, pp. 19–27. doi: 10.1109/ICCV.2015.10

2015 doi
[28]

Layer normalization

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. “Layer normalization”. In: CoRR abs/1607.06450 (2016). arXiv: 1607.06450 [cs.LG]. 153

2016 arXiv
[29]

Deep Residual Learning for Image Recognition

Kaiming He et al. “Deep Residual Learning for Image Recognition”. In: CoRR abs/1512.03385 (2016). arXiv: 1512.03385 [cs.CV]

2016 arXiv
[31]

Gaussian Error Linear Units (GELUs)

Dan Hendrycks and Kevin Gimpel. “Gaussian Error Linear Units (GELUs)”. In: arXiv preprint arXiv:1606.08415 (2016)

2016 arXiv
[32]

Convolutional Neural Net- works Using Logarithmic Data Representation

Daisuke Miyashita, Edward H. Lee, and Boris Murmann. “Convolutional Neural Net- works Using Logarithmic Data Representation”. In:CoRR abs/1603.01025 (2016). arXiv: 1603.01025 [cs.LG]

2016 arXiv
[33]

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Pranav Rajpurkar et al. SQuAD: 100,000+ Questions for Machine Comprehension of Text. 2016. arXiv: 1606.05250 [cs.CL]. url: https://arxiv.org/abs/1606.05250

2016 arXiv
[34]

Regularization with stochastic transformations and perturbations for deep semi-supervised learning

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. “Regularization with stochastic transformations and perturbations for deep semi-supervised learning”. In: Advances in neural information processing systems . 2016, pp. 1163–1171

2016
[35]

Neural machine translation of rare words with subword units

Rico Sennrich, Barry Haddow, and Alexandra Birch. “Neural machine translation of rare words with subword units”. In:Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers . Th...

2016
[36]

Google’s Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation

Yonghui Wu et al. “Google’s Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation”. In: CoRR abs/1609.08144 (2016). arXiv: 1609.08144 [cs.CL]. url: http://arxiv.org/abs/1609.08144

2016 arXiv
[37]

Massive exploration of neural machine translation architectures

Denny Britz et al. “Massive exploration of neural machine translation architectures”. In: CoRR abs/1703.03906 (2017). arXiv: 1703.03906 [cs.CL]

2017 arXiv
[38]

Deep reinforcement learning from human preferences

Paul F. Christiano et al. “Deep reinforcement learning from human preferences”. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 . Ed. by Isabelle Guyon et al. Curran Associates, Inc. Long Beach, CA, USA,...

2017
[39]

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Itay Hubara et al. “Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations”. In: J. Mach. Learn. Res 18 (2017), pp. 6869–6898

2017
[40]

Quantization and Training of Neural Networks for Efficient Integer- Arithmetic-Only Inference

Benoit Jacob et al. Quantization and Training of Neural Networks for Efficient Integer- Arithmetic-Only Inference. 2017. arXiv: 1712.05877 [cs.LG]

2017 arXiv
[41]

Searching for activation func- tions

Prajit Ramachandran, Barret Zoph, and Quoc V. Le. “Searching for activation func- tions”. In: arXiv preprint arXiv:1710.05941 (2017)

2017 arXiv
[42]

Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. Transfer Learning for Se- quence Tagging with Hierarchical Recurrent Networks. 2017. arXiv: 1703.06345 [cs.CL]

2017 arXiv
[43]

Big Data and Machine Learning in Health Care

Andrew L. Beam and Isaac S. Kohane. “Big Data and Machine Learning in Health Care”. In: JAMA 319.13 (2018), pp. 1317–1318

2018
[44]

Deep learning and algorithmic trading

Hans Buehler et al. “Deep learning and algorithmic trading”. In: Financial Markets and Portfolio Management 32.3 (2018), pp. 239–260

2018
[45]

Universal Language Model Fine-tuning for Text Classification

Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. 2018. arXiv: 1801.06146 [cs.CL]. 154

2018 arXiv
[46]

Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

Taku Kudo and John Richardson. “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing”. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations. Ed. by...

2018
[47]

Generalization without systematicity: On the compo- sitional skills of sequence-to-sequence recurrent networks

Brenden Lake and Marco Baroni. “Generalization without systematicity: On the compo- sitional skills of sequence-to-sequence recurrent networks”. In: International Conference on Machine Learning . PMLR. 2018, pp. 2873–2882

2018
[48]

WWW’18 Open Challenge: Fi- nancial Opinion Mining and Question Answering

Macedo Maia, Siegfried Handschuh, Andr´ e Freitas, et al. “WWW’18 Open Challenge: Fi- nancial Opinion Mining and Question Answering”. In:Companion Proceedings of WWW (2018), pp. 1941–1942

2018
[49]

Did the model understand the question?

Pramod Kaushik Mudrakarta et al. “Did the model understand the question?” In: Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne, Australia: Association for Computational Linguis- tics, 2018, pp. 1896–19...

2018 doi
[50]

Peters et al

Matthew E. Peters et al. Deep Contextualized Word Representations . arXiv preprint
[51]

Improving Language Understanding by Generative Pre-training

Alec Radford et al. Improving Language Understanding by Generative Pre-training . Available online. 2018

2018
[52]

Self-attention with relative position representations

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. “Self-attention with relative position representations”. In: CoRR abs/1803.02155 (2018). arXiv: 1803.02155 [cs.CL]

2018 arXiv
[53]

Deep EHR: A survey of recent advances in deep learning tech- niques for electronic health record (EHR) analysis

Benjamin Shickel et al. “Deep EHR: A survey of recent advances in deep learning tech- niques for electronic health record (EHR) analysis”. In: IEEE journal of biomedical and health informatics 22.5 (2018), pp. 1589–1604

2018
[54]

What makes reading comprehension questions easier?

Saku Sugawara et al. “What makes reading comprehension questions easier?” In: Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . Brussels, Belgium: Association for Computational Linguistics, 2018, pp. 4208–4219. doi: 10.18653/v1/D18-1453...

2018 doi
[55]

A Simple Method for Commonsense Reasoning

Trieu H. Trinh and Quoc V. Le. “A Simple Method for Commonsense Reasoning”. In: CoRR abs/1806.02847 (2018). arXiv: 1806.02847 [cs.AI]

2018 arXiv
[56]

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Alex Wang et al. “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding”. In: Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLPEMNLP 2018, Brussels, Belgium, November 1, 2018 . Ed. by Tal Linzen, Grzego...

2018
[57]

Attention? Attention!

Lilian Weng. “Attention? Attention!” In: lilianweng.github.io (2018). url: https : / / lilianweng.github.io/posts/2018-06-24-attention/

2018
[58]

Hybrid deep sequential modeling for social text-driven stock predic- tion

Huizhe Wu et al. “Hybrid deep sequential modeling for social text-driven stock predic- tion”. In: Proceedings of ACM CIKM. 2018, pp. 1627–1630

2018
[59]

Stock movement prediction from tweets and historical prices

Yumo Xu and Shay B Cohen. “Stock movement prediction from tweets and historical prices”. In: Proceedings of ACL. 2018, pp. 1970–1979

2018
[60]

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Ques- tion Answering

Zhilin Yang et al. “HotpotQA: A Dataset for Diverse, Explainable Multi-hop Ques- tion Answering”. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics. Brussels, Belgium, 2018, pp. 2369–2380. 155

2018
[61]

Publicly available clinical BERT embeddings

Emily Alsentzer et al. “Publicly available clinical BERT embeddings”. In: arXiv preprint arXiv:1904.03323 (2019)

2019 arXiv
[62]

Adaptive Input Representations for Neural Language Modeling

Alexei Baevski and Michael Auli. “Adaptive Input Representations for Neural Language Modeling”. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net. 2019

2019
[63]

Single-cell trajectories reconstruction, exploration and mapping of omics data with STREAM

H. Chen et al. “Single-cell trajectories reconstruction, exploration and mapping of omics data with STREAM”. In: Nature Communications 10.1 (2019), p. 1903. doi: 10.1038/ s41467 - 019 - 09670 - 4. url: https : / / www . nature . com / articles / s41467 - 019 - 09670-4

2019
[64]

Generating Long Sequences with Sparse Transformers

Rewon Child et al. “Generating Long Sequences with Sparse Transformers”. In: CoRR abs/1904.10509 (2019). arXiv: 1904.10509 [cs.LG]

2019 arXiv
[65]

Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin et al. “Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding”. In: Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019. Ed. by ...

2019 doi
[66]

Unified Language Model Pre-training for Natural Language Understand- ing and Generation

Li Dong et al. “Unified Language Model Pre-training for Natural Language Understand- ing and Generation”. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC...

2019
[67]

OpenWebText Corpus

Aaron Gokaslan, Ellie Pavlick, and Stefanie Tellex. OpenWebText Corpus . http : / / Skylion007.github.io/OpenWebTextCorpus. 2019

2019
[68]

Parameter-Efficient Transfer Learning for NLP

Neil Houlsby et al. Parameter-Efficient Transfer Learning for NLP . 2019. arXiv: 1902. 00751 [cs.LG]

2019
[69]

PubMedQA: A Dataset for Biomedical Research Question Answer- ing

Qingyu Jin et al. “PubMedQA: A Dataset for Biomedical Research Question Answer- ing”. In: Proceedings of EMNLP-IJCNLP (2019), pp. 2567–2577

2019
[70]

The Personalization of Conversational Agents in Health Care: Systematic Review

A. Baki Kocaballi et al. “The Personalization of Conversational Agents in Health Care: Systematic Review”. In: Journal of Medical Internet Research 21.11 (2019). doi: 10. 2196/15360. url: https://www.jmir.org/2019/11/e15360/

2019
[71]

Natural Questions: A Benchmark for Question Answering Research

Tom Kwiatkowski et al. “Natural Questions: A Benchmark for Question Answering Research”. In: Transactions of the Association for Computational Linguistics 7 (2019). Ed. by Lillian Lee et al., pp. 452–466. doi: 10.1162/tacl\_a\_00276 . url: https: //aclanthology.org/Q19-1026

2019 doi
[72]

Multi-task deep neural networks for natural language understand- ing

Xiaodong Liu et al. “Multi-task deep neural networks for natural language understand- ing”. In: CoRR abs/1901.11504 (2019). arXiv: 1901.11504 [cs.CL]

2019 arXiv
[73]

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yinhan Liu et al. “RoBERTa: A Robustly Optimized BERT Pretraining Approach”. In: arXiv preprint arXiv:1907.11692 . 2019

1907 arXiv
[74]

Jason Phang, Thibault F´ evry, and Samuel R. Bowman. Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks . 2019. arXiv: 1811.01088 [cs.CL]

2019 arXiv
[75]

Language Models Are Unsupervised Multitask Learners

Alec Radford et al. Language Models Are Unsupervised Multitask Learners . 2019. url: https://openai.com/blog/better-language-models/ . 156

2019
[76]

Transfer Learning in Natural Language Processing

Sebastian Ruder et al. “Transfer Learning in Natural Language Processing”. In: Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials . Ed. by Anoop Sarkar and Michael Strube. Min- neapolis, Minnesota: Ass...

2019 doi
[77]

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Victor Sanh et al. “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”. In: Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS (2019), pp. 12–23

2019
[78]

Fast Transformer Decoding: One Write-Head is All You Need

Noam Shazeer. “Fast Transformer Decoding: One Write-Head is All You Need”. In: CoRR abs/1911.02150 (2019). arXiv: 1911.02150 [cs.CL]. url: http://arxiv.org/ abs/1911.02150

2019 arXiv
[79]

Improving fraud detection in financial services through deep learning

Timothy Smith and Manish Kumar. “Improving fraud detection in financial services through deep learning”. In: Journal of Financial Crime 26.4 (2019), pp. 1062–1073

2019
[80]

Energy and Policy Consid- erations for Deep Learning in NLP

Emma Strubell, Ananya Ganesh, and Andrew McCallum. “Energy and Policy Consid- erations for Deep Learning in NLP”. In: ACL 2019. 2019

2019
[81]

CommonsenseQA: A Question Answering Challenge Targeting Com- monsense Knowledge

Alon Talmor et al. CommonsenseQA: A Question Answering Challenge Targeting Com- monsense Knowledge . 2019. arXiv: 1811.00937 [cs.CL]

2019 arXiv
[82]

Defending Against Neural Fake News

Rowan Zellers et al. “Defending Against Neural Fake News”. In: Advances in Neural Information Processing Systems 32 . Ed. by Hanna M. Wallach et al. NeurIPS 2019, December 8-14. Vancouver, BC, Canada: NeurIPS, 2019, pp. 9051–9062

2019
[83]

Root Mean Square Layer Normalization

Biao Zhang and Rico Sennrich. “Root Mean Square Layer Normalization”. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. 2019, pp. 12360–12371

2019
[84]

Deep learning enables rapid identification of potent DDR1 kinase inhibitors

Alex Zhavoronkov et al. “Deep learning enables rapid identification of potent DDR1 kinase inhibitors”. In: Nature Biotechnology 37 (2019), pp. 1038–1040. doi: 10.1038/ d41573-019-00170-0

2019
[85]

Fine-tuning language models from human preferences

Daniel M Ziegler et al. “Fine-tuning language models from human preferences”. In: CoRR abs/1909.08593 (2019)

2019 arXiv
[86]

Towards a Human-like Open-Domain Chatbot

Daniel Adiwardana et al. Towards a Human-like Open-Domain Chatbot . 2020. arXiv: 2001.09977 [cs.CL]

2020 arXiv
[87]

The Pushshift Reddit Dataset

Jason Baumgartner et al. “The Pushshift Reddit Dataset”. In: Proceedings of the Four- teenth International AAAI Conference on Web and Social Media . ICWSM 2020, Held Virtually. Atlanta, Georgia, USA: AAAI Press, 2020, pp. 830–839

2020
[88]

Brown et al

Tom B. Brown et al. Language Models Are Few-Shot Learners. 2020. arXiv: 2005.14165 [cs.CL]

2020 arXiv
[89]

Don ’t Stop Pretraining: Adapt Language Models to Domains and Tasks

Suchin Gururangan et al. Don ’t Stop Pretraining: Adapt Language Models to Domains and Tasks. 2020. arXiv: 2004.10964 [cs.CL]

2020 arXiv
[90]

Scaling Laws for Autoregressive Generative Modeling

Tom Henighan et al. “Scaling Laws for Autoregressive Generative Modeling”. In: arXiv preprint arXiv:2010.14701 (2020)

2020 arXiv
[91]

The Curious Case of Neural Text Degeneration

Ari Holtzman et al. “The Curious Case of Neural Text Degeneration”. In: 8th Inter- national Conference on Learning Representations, ICLR 2020 (2020). OpenReview.net. url: https://openreview.net/forum?id=rygGQyrFvH

2020
[92]

Ethical considerations for AI in finance

Michael Jones et al. “Ethical considerations for AI in finance”. In: AI & Society 35.1 (2020), pp. 287–300. 157

2020
[93]

Scaling Laws for Neural Language Models

Jared Kaplan et al. “Scaling Laws for Neural Language Models”. In: CoRR abs/2001.08361 (2020)

2020 arXiv
[94]

BART: Denoising Sequence-to-Sequence Pre-training for Natural Lan- guage Generation, Translation, and Comprehension

Mike Lewis et al. “BART: Denoising Sequence-to-Sequence Pre-training for Natural Lan- guage Generation, Translation, and Comprehension”. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computa- tional Linguistics. 2...

2020
[95]

MAEC: A Multimodal Aligned Earnings Conference Call Dataset for Financial Risk Prediction

Jiazheng Li et al. “MAEC: A Multimodal Aligned Earnings Conference Call Dataset for Financial Risk Prediction”. In: Proceedings of ACM CIKM. 2020, pp. 3063–3070

2020
[96]

Natural language processing in risk management and compliance

Jin Li, Scott Spangler, and Yue Yu. “Natural language processing in risk management and compliance”. In: Journal of Risk Management in Financial Institutions 13.2 (2020), pp. 158–175

2020
[97]

Understanding the difficulty of training transformers

Lizi Liu et al. “Understanding the difficulty of training transformers”. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020 . 2020, pp. 5747–5763

2020
[98]

The financial document causality detection shared task (fincausal 2020)

Dominique Mariko, Hanna Abi Akl, Estelle Labidurie, et al. “The financial document causality detection shared task (fincausal 2020)”. In: Proceedings of the Workshop on FNP-FNS. 2020, pp. 23–32

2020
[99]

Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer

Colin Raffel et al. “Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer”. In: Journal of Machine Learning Research 21 (2020), 140:1–140:67

2020
[100]

GLU Variants Improve Transformer

Noam Shazeer. “GLU Variants Improve Transformer”. In: arXiv preprint arXiv:2002.05202 (2020)

2020 arXiv
[101]

On Layer Normalization in the Transformer Architecture

Ruibo Xiong et al. “On Layer Normalization in the Transformer Architecture”. In: ICML. 2020

2020
[102]

Big Bird: Transformers for Longer Sequences

Manzil Zaheer et al. “Big Bird: Transformers for Longer Sequences”. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual . 2020

2020
[103]

Muppet: Massive multi-task representations with pre-finetuning

Anna Aghajanyan et al. “Muppet: Massive multi-task representations with pre-finetuning”. In: CoRR abs/2109.08668 (2021). arXiv: 2109.08668 [cs.CL]

2021 arXiv
[104]

A General Language Assistant as a Laboratory for Alignment

Amanda Askell et al. “A General Language Assistant as a Laboratory for Alignment”. In: CoRR abs/2112.00861 (2021)

2021 arXiv
[105]

Program synthesis with large language models

James Austin et al. “Program synthesis with large language models”. In: CoRR abs/2108.07732 (2021)

2021 arXiv
[106]

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

Emily M Bender et al. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” In: F AccT ’21 (2021)

2021
[107]

Extracting training data from large language models

Nicholas Carlini et al. “Extracting training data from large language models”. In: 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021. 2021, pp. 2633– 2650

2021
[108]

Evaluating Large Language Models Trained on Code

Mark Chen et al. Evaluating Large Language Models Trained on Code . arXiv preprint arXiv:2107.03374. 2021

2021 arXiv
[109]

CogView: Mastering Text-to-Image Generation via Transformers

Ming Ding et al. “CogView: Mastering Text-to-Image Generation via Transformers”. In: Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual . 2021, pp. 19822–19835. 158

2021
[110]

Switch Transformers: Scaling to Tril- lion Parameter Models with Simple and Efficient Sparsity

William Fedus, Barret Zoph, and Noam Shazeer. “Switch Transformers: Scaling to Tril- lion Parameter Models with Simple and Efficient Sparsity”. In: J. Mach. Learn. Res (2021), pp. 1–40

2021
[111]

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Leo Gao et al. “The Pile: An 800GB Dataset of Diverse Text for Language Modeling”. In: CoRR abs/2101.00027 (2021). arXiv: 2101.00027 [cs.CL]

2021 arXiv
[112]

Multilingual and cross-lingual intent detection from spoken data

Daniela Gerz et al. “Multilingual and cross-lingual intent detection from spoken data”. In: Proceedings of EMNLP. 2021, pp. 7468–7475

2021
[113]

Measuring Massive Multitask Language Understanding

Dan Hendrycks et al. “Measuring Massive Multitask Language Understanding”. In: Pro- ceedings of the International Conference on Learning Representations (ICLR) . 2021

2021
[114]

Hu et al

Edward J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. 2021. arXiv: 2106.09685 [cs.CL]

2021 arXiv
[115]

Alignment of language agents

Z. Kenton et al. “Alignment of language agents”. In: CoRR abs/2103.14659 (2021)

2021 arXiv
[116]

Efficient sequence packing without cross-contamination: Acceler- ating large language models without impacting performance

Michael M. Krell et al. “Efficient sequence packing without cross-contamination: Acceler- ating large language models without impacting performance”. In: CoRR abs/2107.02027 (2021). arXiv: 2107.02027 [cs.CL]

2021 arXiv
[117]

Why machine reading comprehension models learn shortcuts?

Yuxuan Lai et al. “Why machine reading comprehension models learn shortcuts?” In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 . Online: Association for Computational Linguistics, 2021, pp. 989–1002. doi: 10 . 18653 / v1 / 2021.findings-acl.85. ur...

2021
[118]

The Power of Scale for Parameter- Efficient Prompt Tuning

Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter- Efficient Prompt Tuning. 2021. arXiv: 2104.08691 [cs.CL]

2021 arXiv
[119]

Prefix-Tuning: Optimizing Continuous Prompts for Gen- eration

Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Gen- eration. 2021. arXiv: 2101.00190 [cs.CL]

2021 arXiv
[120]

A survey on deep learning in medical image analysis

Zhi Li, Qiang Zhang, Qi Dou, et al. “A survey on deep learning in medical image analysis”. In: Medical image analysis 67 (2021), p. 101813

2021
[121]

Jurassic-1: Technical details and evaluation

Or Lieber et al. “Jurassic-1: Technical details and evaluation”. In: White Paper. AI21 Labs 1 (2021)

2021
[122]

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Pengfei Liu et al. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing . arXiv preprint arXiv:2107.13586. 2021. url: https://arxiv.org/abs/2107.13586

2021 arXiv
[123]

A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers

Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers . 2021. arXiv: 2106.15772 [cs.AI]

2021 arXiv
[124]

WebGPT: Browser-assisted Question-Answering with Human Feed- back

R. Nakano et al. “WebGPT: Browser-assisted Question-Answering with Human Feed- back”. In: CoRR abs/2112.09332 (2021)

2021 arXiv
[125]

Do Transformer Modifications Transfer Across Implementations and Applications?

Sharan Narang et al. “Do Transformer Modifications Transfer Across Implementations and Applications?” In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 . 2...

2021
[126]

GPT3-toPlan: Extracting Plans from Text using GPT-3

A. Olmo, S. Sreedharan, and S. Kambhampati. “GPT3-toPlan: Extracting Plans from Text using GPT-3”. In: FinPlan 2021 (2021), p. 24

2021
[127]

Enhancing customer service through AI-driven virtual assistants in the banking sector

Arpan Pal, Aniruddha Kundu, and Rajdeep Chakraborty. “Enhancing customer service through AI-driven virtual assistants in the banking sector”. In: Journal of Banking and Financial Technology 5.1 (2021), pp. 1–12

2021
[128]

Random Feature Attention

Baolin Peng, Xiang Li, and Percy Liang. “Random Feature Attention”. In: CoRR abs/2106.14448 (2021). arXiv: 2106.14448 [cs.CL]. 159

2021 arXiv
[129]

Learning how to ask: Querying LMs with mixtures of soft prompts

Guanghui Qin and Jason Eisner. “Learning how to ask: Querying LMs with mixtures of soft prompts”. In: CoRR abs/2104.06599 (2021). arXiv: 2104.06599 [cs.CL]

2021 arXiv
[130]

Learning Transferable Visual Models From Natural Language Super- vision

Alec Radford et al. Learning Transferable Visual Models From Natural Language Super- vision. 2021. arXiv: 2103.00020 [cs.CV]

2021 arXiv
[131]

Scaling language models: Methods, analysis & insights from training Gopher

Jack W. Rae et al. “Scaling language models: Methods, analysis & insights from training Gopher”. In: CoRR abs/2112.11446 (2021). arXiv: 2112.11446 [cs.CL]

2021 arXiv
[132]

Zero-Shot Text-to-Image Generation

Aditya Ramesh et al. Zero-Shot Text-to-Image Generation . 2021. arXiv: 2102 . 12092 [cs.CV]

2021
[133]

Impact of news on the commodity market: Dataset and results

Ankur Sinha and Tanmay Khandait. “Impact of news on the commodity market: Dataset and results”. In: Proceedings of FICC. 2021, pp. 589–601

2021
[134]

RoFormer: Enhanced Transformer with Rotary Position Embedding

Jianlin Su et al. “RoFormer: Enhanced Transformer with Rotary Position Embedding”. In: arXiv preprint arXiv:2104.09864 (2021)

2021 arXiv
[135]

Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models

Alex Tamkin et al. “Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models”. In: arXiv preprint arXiv:2102.02503 (2021)

2021 arXiv
[136]

Long Range Arena: A Benchmark for Efficient Transformers

Yi Tay et al. “Long Range Arena: A Benchmark for Efficient Transformers”. In: CoRR abs/2011.04006 (2021). arXiv: 2011.04006 [cs.CL]

2021 arXiv
[137]

Frozen in Time: Temporal Contextualization for In-Context Learning

Katerina Tsimpoukelli et al. “Frozen in Time: Temporal Contextualization for In-Context Learning”. In: CoRR abs/2109.14867 (2021). arXiv: 2109.14867 [cs.CL]

2021 arXiv
[138]

Milvus: A Purpose-Built Vector Data Management System

J. Wang et al. “Milvus: A Purpose-Built Vector Data Management System”. In: Pro- ceedings of the 2021 International Conference on Management of Data . 2021, pp. 2614– 2627

2021
[139]

Pangu- α: Large-scale autoregressive pretrained Chinese language models with auto-parallel computation

Weihua Zeng et al. “Pangu- α: Large-scale autoregressive pretrained Chinese language models with auto-parallel computation”. In: CoRR abs/2104.12369 (2021). arXiv: 2104. 12369 [cs.CL]

2021 arXiv
[140]

Medical image analysis with artificial intelligence

Jun Zhang et al. “Medical image analysis with artificial intelligence”. In: IEEE Trans- actions on Biomedical Engineering 68.5 (2021), pp. 1375–1379

2021
[141]

Calibrate Before Use: Improving Few-shot Performance of Language Models

Zihao Zhao et al. “Calibrate Before Use: Improving Few-shot Performance of Language Models”. In: Proceedings of the 38th International Conference on Machine Learning . Ed. by Marina Meila and Tong Zhang. Vol. 139. Proceedings of Machine Learning Re- search. PMLR, 2021, pp. 126...

2021
[142]

Global Table Extractor (GTE): A Framework for Joint Table Iden- tification and Cell Structure Recognition Using Visual Context

Xinyi Zheng et al. “Global Table Extractor (GTE): A Framework for Joint Table Iden- tification and Cell Structure Recognition Using Visual Context”. In: Proceedings of the IEEE/CVF WACV. 2021, pp. 697–706

2021
[143]

Trade the event: Corporate events detection for news-based event-driven trading

Zhihan Zhou, Liqian Ma, and Han Liu. “Trade the event: Corporate events detection for news-based event-driven trading”. In: Findings of ACL-IJCNLP . 2021, pp. 2114–2124

2021
[144]

What Learning Algorithm Is In-context Learning? Investigations with Linear Models

E. Aky¨ urek et al. “What Learning Algorithm Is In-context Learning? Investigations with Linear Models”. In: CoRR abs/2211.15661 (2022)

2022 arXiv
[145]

Flamingo: a Visual Language Model for Few-Shot Learn- ing

Jean-Baptiste Alayrac et al. “Flamingo: a Visual Language Model for Few-Shot Learn- ing”. In: Advances in Neural Information Processing Systems . Ed. by Alice H. Oh et al
[146]

OpenAI ChatGPT Generated Literature Re- view: Digital Twin in Healthcare

¨O˘ guzhan Aydın and Emre Karaarslan. “OpenAI ChatGPT Generated Literature Re- view: Digital Twin in Healthcare”. In: SSRN Electronic Journal (2022). Please replace ”number” with the actual abstract number. url: https : / / ssrn . com / abstract = number. 160

2022
[148]

Training a Helpful and Harmless Assistant with Reinforcement Learn- ing from Human Feedback

Yuntao Bai et al. Training a Helpful and Harmless Assistant with Reinforcement Learn- ing from Human Feedback. 2022. arXiv: 2204.05862 [cs.CL]

2022 arXiv
[149]

Visual Prompting via Image Inpainting

Amir Bar et al. “Visual Prompting via Image Inpainting”. In: Advances in Neural In- formation Processing Systems. Vol. 35. 2022, pp. 25005–25017

2022
[150]

Quantifying memorization across neural language models

Nicholas Carlini et al. “Quantifying memorization across neural language models”. In: CoRR abs/2202.12488 (2022). arXiv: 2202.12488 [cs.CL]

2022 arXiv
[151]

Stephanie C. Y. Chan et al. Data Distributional Properties Drive Emergent In-Context Learning in Transformers. 2022. arXiv: 2205.05055 [cs.LG]

2022 arXiv
[152]

Improving In-Context Few-Shot Learning via Self-Supervised Train- ing

Mingda Chen et al. “Improving In-Context Few-Shot Learning via Self-Supervised Train- ing”. In: Proceedings of the 2022 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies . Ed. by Marine Carpuat, Marie-Cathe...

2022
[153]

ConvFinQA: Exploring the Chain of Numerical Reasoning in Con- versational Finance Question Answering

Zhiyu Chen et al. “ConvFinQA: Exploring the Chain of Numerical Reasoning in Con- versational Finance Question Answering”. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 2022, pp. 6279–6292

2022
[154]

FinQA: A Dataset of Numerical Reasoning Over Financial Data

Zhiyu Chen et al. “FinQA: A Dataset of Numerical Reasoning Over Financial Data”. In: (2022). Presumed publication year and citation style as ”2022a”, specifics such as journal name, volume, issue, pages, and DOI are not provided and should be added

2022
[155]

PaLM: Scaling Language Modeling with Pathways

Aakanksha Chowdhery et al. “PaLM: Scaling Language Modeling with Pathways”. In: CoRR abs/2204.02311 (2022)

2022 arXiv
[156]

Scaling Instruction-Finetuned Language Models

H. W. Chung et al. “Scaling Instruction-Finetuned Language Models”. In: CoRR abs/2210.11416 (2022)

2022 arXiv
[157]

Selection-inference: Exploiting large language models for interpretable logical reasoning

A. Creswell, M. Shanahan, and I. Higgins. “Selection-inference: Exploiting large language models for interpretable logical reasoning”. In: CoRR abs/2205.09712 (2022)

2022 arXiv
[158]

Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers

D. Dai et al. “Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers”. In: (2022)

2022
[160]

Hungry Hungry Hippos: Towards Language Modeling with State Space Models

Tri Dao et al. “Hungry Hungry Hippos: Towards Language Modeling with State Space Models”. In: CoRR abs/2212.14052 (2022). doi: 10.48550/arXiv.2212.14052 . url: https://doi.org/10.48550/arXiv.2212.14052

2022 doi arXiv
[161]

GLAM: Efficient Scaling of Language Models with Mixture-of-Experts

Nan Du et al. “GLAM: Efficient Scaling of Language Models with Mixture-of-Experts”. In: International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bal- timore, Maryland, USA . 2022, pp. 5547–5569

2022
[162]

How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources

Hao Fu Yao; Peng and Tushar Khot. “How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources”. In: Yao Fu’s Notion (2022). url: \url{"https://yaofu.notion.site/How- does- GPT- Obtain- its- Ability- Tracing-Emergent-Abilities-of-Language-Mo...

2022
[163]

Complexity-based prompting for multi-step reasoning

Y. Fu et al. “Complexity-based prompting for multi-step reasoning”. In: CoRR abs/2210.00720 (2022)

2022 arXiv
[164]

PAL: program-aided language models

L. Gao et al. “PAL: program-aided language models”. In: CoRR abs/2211.10435 (2022)

2022 arXiv
[165]

Improving Alignment of Dialogue Agents via Targeted Human Judge- ments

A. Glaese et al. “Improving Alignment of Dialogue Agents via Targeted Human Judge- ments”. In: CoRR abs/2209.14375 (2022)

2022 arXiv
[166]

Demystifying Prompts in Language Models via Perplexity Estimation

Hila Gonen et al. Demystifying Prompts in Language Models via Perplexity Estimation
[167]

Efficiently Modeling Long Sequences with Structured State Spaces

Albert Gu, Karan Goel, and Christopher R´ e. “Efficiently Modeling Long Sequences with Structured State Spaces”. In: The Tenth International Conference on Learning Representations. Accessed: 2024-04-13. 2022. url: https://openreview.net/forum? id=uYLFoz1vlAC

2024
[169]

arXiv: 2212.04037 [cs.CL]

arXiv
[170]

Towards a Unified View of Parameter-Efficient Transfer Learning

Junxian He et al. Towards a Unified View of Parameter-Efficient Transfer Learning
[171]

Scaling laws and interpretability of learning from repeated data

Daniel Hernandez et al. “Scaling laws and interpretability of learning from repeated data”. In: CoRR abs/2205.10487 (2022). arXiv: 2205.10487 [cs.LG]

2022 arXiv
[173]

url: https://arxiv.org/abs/2206.06336

arXiv
[174]

OPT-IML: Scaling Language Model Instruction Meta Learning Through the Lens of Generalization

Srinivasan Iyer et al. “OPT-IML: Scaling Language Model Instruction Meta Learning Through the Lens of Generalization”. In: CoRR abs/2212.12017 (2022). arXiv: 2212. 12017 [cs.CL]

2022 arXiv
[175]

arXiv: 2110.04366 [cs.CL]

arXiv
[176]

Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator

H. J. Kim et al. “Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator”. In: CoRR abs/2206.08082 (2022)

2022 arXiv
[177]

Training Compute-Optimal Large Language Models

Jan Hoffmann et al. “Training Compute-Optimal Large Language Models”. In: CoRR abs/2203.15556 (2022)

2022 arXiv
[178]

Instruction Induction: From Few Examples to Natural Language Task Descriptions

Or Honovich et al. Instruction Induction: From Few Examples to Natural Language Task Descriptions. 2022. arXiv: 2205.10782 [cs.CL]

2022 arXiv
[179]

The BigScience ROOTS Corpus: A 1.6 TB Composite Multi- lingual Dataset

Herv´ e Lauren¸ con et al. “The BigScience ROOTS Corpus: A 1.6 TB Composite Multi- lingual Dataset”. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track . NeurIPS. 2022

2022
[180]

Decomposed prompting: A modular approach for solving complex tasks

T. Khot et al. “Decomposed prompting: A modular approach for solving complex tasks”. In: CoRR abs/2210.02406 (2022). https://doi.org/10.48550/arXiv.2210.02406

2022 doi arXiv
[181]

Deduplicating training data makes language models better

Kenton Lee et al. “Deduplicating training data makes language models better”. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 . 2022, pp. 8424– 8445

2022
[182]

Replace Grammarly Premium with OpenAI ChatGPT

Sung Kim. Replace Grammarly Premium with OpenAI ChatGPT . 2022. url: https: //medium.com/geekculture/replace-grammarly-premium-with-openai-chatgpt- 320049179c79

2022
[183]

BioASQ-QA: A manually curated corpus for biomedical ques- tion answering

Anastasia Krithara et al. BioASQ-QA: A manually curated corpus for biomedical ques- tion answering . 2022

2022
[184]

Competition-level code generation with AlphaCode

Y. Li et al. “Competition-level code generation with AlphaCode”. In: Science (2022)

2022
[185]

What language model to train if you have one million GPU hours?

Teven Le Scao et al. “What language model to train if you have one million GPU hours?” In: Findings of the Association for Computational Linguistics: EMNLP 2022 . Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022, pp. 765–782. url: https://aclant...

2022
[186]

What makes good in-context examples for gpt-3?

J. Liu et al. “What makes good in-context examples for gpt-3?” In: Proceedings of Deep Learning Inside Out (DeeLIO): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, at ACL 2022 . Dublin, Ireland and Online, 2022, pp. 100–114

2022
[187]

Solving quantitative reasoning problems with language models

Aitor Lewkowycz et al. “Solving quantitative reasoning problems with language models”. In: CoRR abs/2206.14858 (2022)

2022 arXiv
[188]

P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks

Xiang Lisa Li and Percy Liang. “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks”. In: CoRR abs/2202.12108 (2022). arXiv: 2202.12108 [cs.CL]

2022 arXiv
[189]

Finer: Financial numeric entity recognition for xbrl tagging

Lefteris Loukas et al. “Finer: Financial numeric entity recognition for xbrl tagging”. In: Proceedings of ACL (2022), pp. 4419–4431

2022
[190]

Holistic Evaluation of Language Models

Percy Liang et al. Holistic Evaluation of Language Models. 2022. doi: 10.48550/arXiv. 2211.09110. url: https://doi.org/10.48550/arXiv.2211.09110

2022 doi arXiv
[191]

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

Renqian Luo et al. “BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining”. In: Briefings in Bioinformatics 23.6 (2022). doi: 10.1093/ bib/bbac409. url: https://doi.org/10.1093\%2Fbib\%2Fbbac409

2022
[192]

Fast and Memory-Efficient Attention with FlashAttention-2

Lizi Liu et al. “Fast and Memory-Efficient Attention with FlashAttention-2”. In: CoRR abs/2205.14135 (2022). arXiv: 2205.14135 [cs.LG]

2022 arXiv
[193]

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Uni- versally Across Scales and Tasks

Xiao Liu et al. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Uni- versally Across Scales and Tasks . 2022. arXiv: 2110.07602 [cs.CL]

2022 arXiv
[194]

Monopoly: Financial prediction from monetary policy conference videos using multimodal cues

Puneet Mathur et al. “Monopoly: Financial prediction from monetary policy conference videos using multimodal cues”. In: Proceedings of ACM MM. 2022, pp. 2276–2285

2022
[195]

Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

Y. Lu et al. “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity”. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland, 2022, pp. 8086– 8098

2022
[196]

MetaICL: Learning to Learn In Context

Sewon Min et al. “MetaICL: Learning to Learn In Context”. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Ed. by Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir ...

2022
[197]

Text and patterns: For effective chain of thought, it takes two to tango

Aman Madaan and Alireza Yazdanbakhsh. “Text and patterns: For effective chain of thought, it takes two to tango”. In: CoRR abs/2209.07686 (2022). arXiv: 2209.07686 [cs.CL]

2022 arXiv
[199]

Cross-task generalization via natural language crowdsourcing instructions

Swaroop Mishra et al. “Cross-task generalization via natural language crowdsourcing instructions”. In: Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers) . Ed. by Smaranda Muresan, Preslav Nakov, and Aline Villavic...

2022
[200]

Long Range Language Modeling via Gated State Spaces

Hrushikesh Mehta et al. “Long Range Language Modeling via Gated State Spaces”. In: CoRR abs/2206.13947 (2022). doi: 10 . 48550 / arXiv . 2206 . 13947. url: https : //doi.org/10.48550/arXiv.2206.13947

2022 doi arXiv
[201]

ECTSum: A New Benchmark Dataset for Bullet Point Sum- marization of Long Earnings Call Transcripts

Rajdeep Mukherjee et al. “ECTSum: A New Benchmark Dataset for Bullet Point Sum- marization of Long Earnings Call Transcripts”. In:Proceedings of EMNLP. 2022, pp. 10893– 10906

2022
[202]

Noisy Channel Language Model Prompting for Few-Shot Text Clas- sification

Sewon Min et al. “Noisy Channel Language Model Prompting for Few-Shot Text Clas- sification”. In: Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers) . Ed. by Smaranda Muresan, Preslav Nakov, and Aline Villavicencio...

2022 doi
[203]

Rethinking the Role of Demonstrations: What Makes In-context Learning Work?

Sewon Min et al. “Rethinking the Role of Demonstrations: What Makes In-context Learning Work?” In: CoRR abs/2202.12837 (2022). url: https://arxiv.org/abs/ 2202.12837

2022 arXiv
[204]

In-context Learning and Induction Heads

Catherine Olsson et al. In-context Learning and Induction Heads . 2022. arXiv: 2209. 11895 [cs.LG]

2022
[205]

Crosslingual Generalization Through Multitask Finetuning

Niklas Muennighoff et al. “Crosslingual Generalization Through Multitask Finetuning”. In: CoRR abs/2211.01786 (2022). url: https://arxiv.org/abs/2211.01786

2022 arXiv
[206]

Train Short, Test Long: Attention with Lin- ear Biases Enables Input Length Extrapolation

Ofir Press, Noah A. Smith, and Mike Lewis. “Train Short, Test Long: Attention with Lin- ear Biases Enables Input Length Extrapolation”. In:The Tenth International Conference on Learning Representations. Accessed: 2024-04-13. 2022. url: https://openreview. net/forum?id=JZJ9Zz1vZ6

2024
[207]

Law informs code: A legal informatics approach to aligning artificial intel- ligence with humans

J. J. Nay. “Law informs code: A legal informatics approach to aligning artificial intel- ligence with humans”. In: CoRR abs/2209.13020 (2022). arXiv: 2209.13020 [cs.CY]. url: https://arxiv.org/abs/2209.13020

2022 arXiv
[208]

CodeGen: An Open Large Language Model for Code with Multi- turn Program Synthesis

Erik Nijkamp et al. “CodeGen: An Open Large Language Model for Code with Multi- turn Program Synthesis”. In: arXiv preprint arXiv:2203.13474 (2022)

2022 arXiv
[209]

Multitask prompted training enables zero-shot task generalization

Victor Sanh et al. “Multitask prompted training enables zero-shot task generalization”. In: The Tenth International Conference on Learning Representations, ICLR 2022, Vir- tual Event, April 25-29, 2022 (2022). OpenReview.net

2022
[210]

Training Language Models to Follow Instructions with Human Feed- back

L. Ouyang et al. “Training Language Models to Follow Instructions with Human Feed- back”. In: CoRR abs/2203.02155 (2022)

2022 arXiv
[211]

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

Seongjin Shin et al. “On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model”. In: Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Ed. by Ma...

2022
[212]

Limitations of Language Models in Arithmetic and Symbolic Induction

Jing Qian et al. Limitations of Language Models in Arithmetic and Symbolic Induction
[213]

arXiv: 2208.05051 [cs.CL]

arXiv
[214]

Learning to retrieve prompts for in-context learn- ing

O. Rubin, J. Herzig, and J. Berant. “Learning to retrieve prompts for in-context learn- ing”. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) . Seattle, W A, United State...

2022
[215]

An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels

Taylor Sorensen et al. “An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels”. In: Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers) . Ed. by Smaranda Muresan, Preslav Nakov, and Ali...

2022 doi
[216]

Finred: A dataset for relation extraction in financial domain

Soumya Sharma et al. “Finred: A dataset for relation extraction in financial domain”. In: Companion Proceedings of WWW. 2022, pp. 595–597. 164

2022
[217]

ChatGPT: The end of online exam integrity?

T. Susnjak. “ChatGPT: The end of online exam integrity?” In: CoRR abs/2212.09292 (2022). url: https://arxiv.org/abs/2212.09292

2022 arXiv
[218]

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Mirac Suzgun et al. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them . 2022. arXiv: 2210.09261 [cs.CL]

2022 arXiv
[219]

ProgPrompt: Generating Situated Robot Task Plans Using Large Language Models

Ishaan Singh et al. “ProgPrompt: Generating Situated Robot Task Plans Using Large Language Models”. In: CoRR abs/2209.11302 (2022)

2022 arXiv
[220]

Large language models encode clinical knowledge

Karan Singhal et al. “Large language models encode clinical knowledge”. In: arXiv preprint arXiv:2212.13138 (2022)

2022 arXiv
[221]

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Samyam Smith et al. “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”. In: CoRR abs/2201.11990 (2022)

2022 arXiv
[222]

Legal prompt engineering for multilingual legal judgement prediction

Daniel Trautmann, Aleksandra Petrova, and Frank Schilder. “Legal prompt engineering for multilingual legal judgement prediction”. In: CoRR abs/2212.02199 (2022). arXiv: 2212.02199. url: https://arxiv.org/abs/2212.02199

2022 arXiv
[223]

Accurate stock movement prediction with self-supervised learning from sparse noisy tweets

Yejun Soun et al. “Accurate stock movement prediction with self-supervised learning from sparse noisy tweets”. In: IEEE Big Data . 2022, pp. 1691–1700

2022
[224]

Iteratively Prompt Pre-trained Language Models for Chain of Thought

Boshi Wang, Xiang Deng, and Huan Sun. “Iteratively Prompt Pre-trained Language Models for Chain of Thought”. In: Proceedings of The 2022 Conference on Empirical Methods for Natural Language Processing (EMNLP) . Online and in-person event, 2022

2022
[225]

DeepNet: Scaling Transformers to 1,000 Layers

Haoyuan Wang et al. “DeepNet: Scaling Transformers to 1,000 Layers”. In: CoRR abs/2203.00555 (2022). arXiv: 2203.00555 [cs.CL]

2022 arXiv
[226]

MVP: Multi-Task Supervised Pre-training for Natural Language Gen- eration

Tian Tang et al. “MVP: Multi-Task Supervised Pre-training for Natural Language Gen- eration”. In: CoRR abs/2206.12131 (2022). arXiv: 2206.12131 [cs.CL]

2022 arXiv
[227]

Galactica: A Large Language Model for Science

Ross Taylor et al. Galactica: A Large Language Model for Science . http://arxiv.org/ abs/2211.09085. arXiv:2211.09085. Nov. 2022

2022 arXiv
[228]

LaMDA: Language Models for Dialog Applications

Romal Thoppilan et al. “LaMDA: Language Models for Dialog Applications”. In: CoRR abs/2201.08239 (2022). arXiv: 2201.08239 [cs.CL]

2022 arXiv
[229]

Super-NaturalInstructions: Generalization via Declarative Instruc- tions on 1600+ NLP Tasks

Yada Wang et al. “Super-NaturalInstructions: Generalization via Declarative Instruc- tions on 1600+ NLP Tasks”. In: CoRR abs/2209.13107 (2022). arXiv: 2209 . 13107 [cs.CL]

2022 arXiv
[230]

Interleaving retrieval with chain-of-thought reasoning for knowledge- intensive multi-step questions

H. Trivedi et al. “Interleaving retrieval with chain-of-thought reasoning for knowledge- intensive multi-step questions”. In: arXiv preprint arXiv:2212.10509 (2022)

2022 arXiv
[231]

Fine-tuned Language Models are Zero-shot Learners

J. Wei et al. “Fine-tuned Language Models are Zero-shot Learners”. In: The Tenth International Conference on Learning Representations, ICLR 2022 . OpenReview.net. Virtual Event, 2022

2022
[232]

Emergent Abilities of Large Language Models

Jason Wei et al. Emergent Abilities of Large Language Models. 2022. arXiv: 2206.07682 [cs.CL]

2022 arXiv
[233]

Rationale-augmented ensembles in language models

X. Wang et al. “Rationale-augmented ensembles in language models”. In: CoRR abs/2206.02336 (2022)

2022 arXiv
[234]

Self-consistency improves chain of thought reasoning in language mod- els

X. Wang et al. “Self-consistency improves chain of thought reasoning in language mod- els”. In: arXiv preprint arXiv:2203.11171 (2022). 165

2022 arXiv
[235]

Self-Instruct: Aligning Language Model with Self Generated Instruc- tions

Yada Wang et al. “Self-Instruct: Aligning Language Model with Self Generated Instruc- tions”. In: CoRR abs/2212.10560 (2022)

2022 arXiv
[236]

React: Synergizing reasoning and acting in language models

S. Yao et al. “React: Synergizing reasoning and acting in language models”. In: CoRR abs/2210.03629 (2022)

2022 arXiv
[237]

Chain of thought prompting elicits reasoning in large language models

J. Wei et al. “Chain of thought prompting elicits reasoning in large language models”. In: CoRR abs/2201.11903 (2022)

2022 arXiv
[238]

Generate rather than retrieve: Large language models are strong context generators

W. Yu et al. “Generate rather than retrieve: Large language models are strong context generators”. In: arXiv preprint arXiv:2209.10063 (2022)

2022 arXiv
[239]

GLM-130B: An Open Bilingual Pre-trained Model

Ailing Zeng et al. GLM-130B: An Open Bilingual Pre-trained Model. 2022. arXiv: 2210. 02414 [cs.CL]

2022
[240]

Self-Adaptive In-Context Learning

Zhiyong Wu et al. “Self-Adaptive In-Context Learning”. In: (2022). Provide additional details such as the journal name, volume, issue, pages, and DOI if available

2022
[241]

An Explanation of In-context Learning as Implicit Bayesian Inference

Sang Michael Xie et al. “An Explanation of In-context Learning as Implicit Bayesian Inference”. In: International Conference on Learning Representations. 2022. url: https: //openreview.net/forum?id=RdJVFCHjUMI

2022
[242]

A Systematic Evaluation of Large Language Models of Code

Frank F. Xu et al. “A Systematic Evaluation of Large Language Models of Code”. In: MAPSPLDI. 2022

2022
[243]

Automatic chain of thought prompting in large language models

Z. Zhang et al. “Automatic chain of thought prompting in large language models”. In: CoRR abs/2210.03493 (2022)

2022 arXiv
[244]

Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations

Kang Min Yoo et al. Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations. 2022. arXiv: 2205.12685 [cs.CL]

2022 arXiv
[245]

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

Joshua Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. 2023. arXiv: 2305.13245 [cs.CL]

2023 arXiv
[246]

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Aarohi Srivastava et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models . 2023. arXiv: 2206.04615 [cs.CL]

2023 arXiv
[247]

Examining scaling and transfer of language model architectures for machine translation

Biao Zhang et al. “Examining scaling and transfer of language model architectures for machine translation”. In: International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA . 2022, pp. 26176–26192

2022
[248]

OPT: open pre-trained transformer language models

Sheng Zhang et al. “OPT: open pre-trained transformer language models”. In: CoRR abs/2205.01068 (2022)

2022 arXiv
[249]

Active Example Selection for In-Context Learning

Yiming Zhang, Shi Feng, and Chenhao Tan. Active Example Selection for In-Context Learning. 2022. arXiv: 2211.04486 [cs.CL]

2022 arXiv
[250]

Self-RAG: Learning to retrieve, generate, and critique through self- reflection

A. Asai et al. “Self-RAG: Learning to retrieve, generate, and critique through self- reflection”. In: arXiv preprint arXiv:2310.11511 (2023)

2023 arXiv
[251]

Least-to-most prompting enables complex reasoning in large language models

D. Zhou et al. “Least-to-most prompting enables complex reasoning in large language models”. In: CoRR abs/2205.10625 (2022)

2022 arXiv
[252]

What Is Semi-Supervised Learning? IBM

Dave Bergmann. What Is Semi-Supervised Learning? IBM. 2023. url: https://www. ibm.com/cloud/learn/semi-supervised-learning (visited on 12/12/2023)

2023
[253]

Can GPT-3 per- form statutory reasoning?

Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. “Can GPT-3 per- form statutory reasoning?” In: CoRR abs/2302.06100 (2023). arXiv: 2302.06100. url: https://arxiv.org/abs/2302.06100

2023 arXiv
[254]

StarCoder: may the source be with you! 2023

Raymond Li et al. StarCoder: may the source be with you! 2023. arXiv: 2305.06161 [cs.CL]. 166

2023 arXiv
[255]

Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPT

M. M. Amin, E. Cambria, and B. W. Schuller. “Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPT”. In: CoRR abs/2303.03186 (2023). arXiv: 2303.03186 [cs.CL]. url: https://arxiv.org/abs/ 2303.03186

2023 arXiv
[256]

How Do In-context Examples Affect Compositional Generaliza- tion?

Shengnan An et al. “How Do In-context Examples Affect Compositional Generaliza- tion?” In: CoRR abs/2305.04835 (2023). url: https://arxiv.org/abs/2305.04835

2023 arXiv
[257]

Data-Juicer: A One-Stop Data Processing System for Large Language Models

Dong Chen et al. “Data-Juicer: A One-Stop Data Processing System for Large Language Models”. In: arXiv preprint arXiv:2305.13169 (2023)

2023 arXiv
[258]

ChatGPT is a Remarkable Tool – For Experts

A. Azaria, R. Azoulay, and S. Reches. “ChatGPT is a Remarkable Tool – For Experts”. In: CoRR abs/2306.03102 (2023). arXiv: 2306.03102 [cs.CL]. url: https://arxiv. org/abs/2306.03102

2023 arXiv
[259]

Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

Wenhu Chen et al. “Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks”. In: (2023). arXiv: 2211 . 12588 [cs.CL]. url: https://arxiv.org/abs/2211.12588

2023 arXiv
[260]

Chatcot: Tool-augmented chain-of-thought reasoning on chat-based large language models

Z. Chen et al. “Chatcot: Tool-augmented chain-of-thought reasoning on chat-based large language models”. In: CoRR abs/2305.14323 (2023)

2023 arXiv
[261]

BioMedLM

Elliot Bolton et al. BioMedLM. https://github.com/stanford-crfm/BioMedLM. 2023

2023
[262]

Academic Writing with GPT-3.5: Reflections on Practices, Efficacy and Transparency

O. O. Buruk. “Academic Writing with GPT-3.5: Reflections on Practices, Efficacy and Transparency”. In: CoRR abs/2304.11079 (2023). arXiv: 2304 . 11079 [cs.CL]. url: https://arxiv.org/abs/2304.11079

2023 arXiv
[263]

Instruction Mining: When Data Mining Meets Large Language Model Finetuning

Yihan Cao et al. Instruction Mining: When Data Mining Meets Large Language Model Finetuning. 2023. arXiv: 2307.06290 [cs.CL]

2023 arXiv
[264]

ChatGPT goes to law school

Jinho H. Choi et al. “ChatGPT goes to law school”. In: (2023). Accessed: 2024-02-14. url: https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=number

2023
[265]

Maybe Only 0.5% Data Is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning

H. Chen et al. “Maybe Only 0.5% Data Is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning”. In: arXiv preprint arXiv:2305.09246 (2023)

2023 arXiv
[266]

Faith and fate: Limits of transformers on compositionality

Nouha Dziri et al. “Faith and fate: Limits of transformers on compositionality”. In: Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) . 2023. url: https://openreview.net/forum?id=Fkckkr3ya8. 167

2023
[267]

A Closer Look at Large Language Models: Emergent Abilities

Yao Fu. A Closer Look at Large Language Models: Emergent Abilities . https://www. notion . so / yaofu / A - Closer - Look - at - Large - Language - Models - Emergent - Abilities-493876b55df5479d80686f68a1abd72f. Accessed: 2023-07-14. 2023

2023
[268]

Is GPT-4 a Good Data Analyst?

Long Cheng, Xiang Li, and Lidong Bing. “Is GPT-4 a Good Data Analyst?” In: CoRR abs/2305.15038 (2023). arXiv: 2305.15038 [cs.LG]. url: https://arxiv.org/abs/ 2305.15038

2023 arXiv
[269]

Lift yourself up: Retrieval-augmented text generation with self mem- ory

X. Cheng et al. “Lift yourself up: Retrieval-augmented text generation with self mem- ory”. In: arXiv preprint arXiv:2305.02437 (2023)

2023 arXiv
[270]

Chiang et al

W.-L. Chiang et al. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%∗Chat- GPT Quality . [Online]. Available: https://vicuna.lmsys.org. 2023

2023
[271]

Leveraging pre-trained large language models to construct and utilize world models for model-based task planning

Lin Guan et al. “Leveraging pre-trained large language models to construct and utilize world models for model-based task planning”. In: Thirty-seventh Conference on Neural Information Processing Systems (2023). url: https://openreview.net/forum?id= zDbsSscmuj

2023
[272]

A Survey on In-context Learning

Qingxiu Dong et al. A Survey on In-context Learning. 2023. arXiv: 2301.00234 [cs.CL]

2023 arXiv
[273]

A Theory of Emergent In-Context Learning as Implicit Structure Induction

Michael Hahn and Navin Goyal. A Theory of Emergent In-Context Learning as Implicit Structure Induction. 2023. arXiv: 2303.07971 [cs.CL]

2023 arXiv
[274]

Using ChatGPT to Conduct a Literature Review

Micha l Haman and Marcin Skolnik. “Using ChatGPT to Conduct a Literature Review”. In: Accountability in Research (2023)

2023
[275]

What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

Shivam Garg et al. What Can Transformers Learn In-Context? A Case Study of Simple Function Classes. 2023. arXiv: 2208.01066 [cs.CL]

2023 arXiv
[276]

Large language models are not abstract reasoners

Guillaume Gendron et al. “Large language models are not abstract reasoners”. In: arXiv preprint arXiv:2305.19555 (2023)

2023 arXiv
[278]

Language Is Not All You Need: Aligning Perception with Language Models

Shaohan Huang et al. Language Is Not All You Need: Aligning Perception with Language Models. arXiv preprint arXiv:2302.14045. 2023. url: https://arxiv.org/abs/2302. 14045

2023 arXiv
[279]

Artificial Hallucinations in ChatGPT: Implications in Scien- tific Writing

S. I. M. Hussam Alkaissi. “Artificial Hallucinations in ChatGPT: Implications in Scien- tific Writing”. In: PubMed (2023). Available on PubMed. url: https://pubmed.ncbi. nlm.nih.gov/ARTICLE\_ID

2023
[280]

How close is ChatGPT to human experts? Comparison corpus, eval- uation, and detection

Binbin Guo et al. “How close is ChatGPT to human experts? Comparison corpus, eval- uation, and detection”. In: CoRR abs/2301.07597 (2023). arXiv: 2301.07597 [cs.CL]

2023 arXiv
[281]

Towards Better Instruction Following Language Models for Chi- nese: Investigating the Impact of Training Data and Evaluation

Yuxuan Ji et al. “Towards Better Instruction Following Language Models for Chi- nese: Investigating the Impact of Training Data and Evaluation”. In: arXiv preprint arXiv:2304.07854 (2023)

2023 arXiv
[282]

MultiFin: A Dataset for Multilingual Financial NLP

Rasmus Jørgensen et al. “MultiFin: A Dataset for Multilingual Financial NLP”. In: Find- ings of the European Chapter of the Association for Computational Linguistics (EACL) . 2023, pp. 864–879

2023
[283]

Reasoning with language model is planning with world model

S. Hao et al. “Reasoning with language model is planning with world model”. In: CoRR abs/2305.14992 (2023)

2023 arXiv
[284]

ChatGPT as Your Personal Data Scientist

Md Mahadi Hassan, Richard A. Knipper, and Shakked K. K. Santu. “ChatGPT as Your Personal Data Scientist”. In: CoRR abs/2305.13657 (2023). arXiv: 2305.13657 [cs.LG]. url: https://arxiv.org/abs/2305.13657

2023 arXiv
[285]

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine- Tuning of Large Language Models

Zhiqiang Hu et al. LLM-Adapters: An Adapter Family for Parameter-Efficient Fine- Tuning of Large Language Models . 2023. arXiv: 2304.01933 [cs.CL]

2023 arXiv
[286]

OpenAssistant Conversations–Democratizing Large Language Model Alignment

Andreas Kopf et al. “OpenAssistant Conversations–Democratizing Large Language Model Alignment”. In: arXiv preprint arXiv:2304.07327 (2023)

2023 arXiv
[287]

Theory of Mind May Have Spontaneously Emerged in Large Language Models

M. Kosinski. “Theory of Mind May Have Spontaneously Emerged in Large Language Models”. In: CoRR abs/2302.02083 (2023). arXiv: 2302.02083 [cs.CL]. url: https: //arxiv.org/abs/2302.02083

2023 arXiv
[288]

I. Ilin. Advanced RAG Techniques: An Illustrated Overview. Accessed: 2024-12-24. 2023. url: https://pub.towardsai.net/advanced- rag- techniques- an- illustrated- overview-04d193d8fec6

2024
[289]

StockEmotions: Discover Investor Emotions for Financial Sentiment Analysis and Multivariate Time Series

Jean Lee et al. “StockEmotions: Discover Investor Emotions for Financial Sentiment Analysis and Multivariate Time Series”. In: AAAI-24 Bridge . 2023

2023
[290]

Contextual Prompting for In-Context Learning

Mukai Li et al. “Contextual Prompting for In-Context Learning”. In: arXiv preprint arXiv:2302.04931 (2023)

2023 arXiv
[291]

On the role of large language models in planning

Subbarao Kambhampati et al. “On the role of large language models in planning”. In: arXiv preprint arXiv:2307.00000 (2023)

2023
[292]

Tree of clarifications: Answering ambiguous questions with retrieval- augmented large language models

G. Kim et al. “Tree of clarifications: Answering ambiguous questions with retrieval- augmented large language models”. In: arXiv preprint arXiv:2310.14696 (2023). 168

2023 arXiv
[293]

Large Language Models are Zero-Shot Reasoners

Takeshi Kojima et al. Large Language Models are Zero-Shot Reasoners . 2023. arXiv: 2205.11916 [cs.CL]

2023 arXiv
[296]

Understanding In-Context Learning

Stanford AI Lab. Understanding In-Context Learning. 2023. url: https://ai.stanford. edu/blog/understanding-incontext/

2023
[299]

Chain of knowledge: A framework for grounding large language models with structured knowledge bases

X. Li et al. “Chain of knowledge: A framework for grounding large language models with structured knowledge bases”. In: arXiv preprint arXiv:2305.13269 (2023)

2023 arXiv
[300]

Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks

Xianzhi Li et al. Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks . 2023. arXiv: 2305.05862 [cs.CL]

2023 arXiv
[2018]

url: https://arxiv.org/abs/1802.05365

arXiv
[2022]

url: https://openreview.net/forum?id=EbMuimAbPbs
[2023]

url: https://arxiv.org/abs/2305.09137

arXiv
[5186]

url: https://aclanthology.org/ 2022.naacl-main.380

doi: 10.18653/v1/2022.naacl-main.380 . url: https://aclanthology.org/ 2022.naacl-main.380

2022 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

More is Different: Broken Symmetry and the Nature of the Hier- archical Structure of Science

Philip W. Anderson. “More is Different: Broken Symmetry and the Nature of the Hier- archical Structure of Science”. In: (1972). url: http://www.lanais.famaf.unc.edu. ar/cursos/em/Anderson-MoreDifferent-1972.pdf

1972

[2] [2]

Mitchell

Tom M. Mitchell. Machine Learning. McGraw-Hill, 1997

1997

[3] [3]

Statistical Learning Theory

Vladimir Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998

1998

[4] [4]

Transductive inference for text classification using support vector machines

Thorsten Joachims. “Transductive inference for text classification using support vector machines”. In: ICML. Citeseer. 1999

1999

[5] [5]

Conditional random fields: Probabilistic models for segmenting and labeling sequence data

John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. “Conditional random fields: Probabilistic models for segmenting and labeling sequence data”. In: Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001) . Ed. by Carla E. Brodley and Andrea P. Danyluk. Morgan Kaufmann, 2001, pp. 282–289

2001

[6] [6]

A Neural Probabilistic Language Model

Yoshua Bengio et al. “A Neural Probabilistic Language Model”. In: Journal of Machine Learning Research 3 (2003), pp. 1137–1155

2003

[7] [7]

V AL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL

R. Howey, D. Long, and M. Fox. “V AL: Automatic plan validation, continuous effects and mixed initiative planning using PDDL”. In: 16th IEEE International Conference on Tools with Artificial Intelligence (2004), pp. 294–301

2004

[8] [8]

Conceptnet–a practical commonsense reasoning tool-kit

Hugo Liu and Push Singh. “Conceptnet–a practical commonsense reasoning tool-kit”. In: BT technology journal 22 (2004), pp. 211–226

2004