REVIEW 3 cited by
Formal Aspects of Language Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models have become one of the most commonly deployed NLP inventions. In the past half-decade, their integration into core natural language processing tools has dramatically increased the performance of such tools, and they have entered the public discourse surrounding artificial intelligence. Consequently, it is important for both developers and researchers alike to understand the mathematical foundations of large language models, as well as how to implement them. These notes are the accompaniment to the theoretical portion of the ETH Z\"urich course on large language models, covering what constitutes a language model from a formal, theoretical perspective.
Forward citations
Cited by 3 Pith papers
-
Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction
Under the chain rule of probability, autoregressive models and energy-based models are in exact bijection in function space, making the global optimum of teacher forcing equivalent to an energy-based model with implic...
-
Unraveling Syntax: Language Modeling and the Substructure of Grammars
Language-modeling loss decomposes linearly over the sub-grammars of a probabilistic context-free grammar, and models learn these sub-grammars in parallel rather than in stages.
-
Circuit Stability Characterizes Language Model Generalization
Circuit stability, measured as rank correlation between soft circuits across subtasks, is proposed as a predictor of language model generalization.
Discussion (0). Continue with ORCID to comment.