REVIEW 3 major objections 5 minor 2 references
How Large Language Models (LLMs) Extrapolate: From Guided Missiles to Guided Prompts
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Large language models are machines of extrapolation; hallucinated references are statistically probable continuations beyond the training data.
desk verdict A genuine historical genealogy of extrapolation, with a central conceptual claim that is more slogan than demonstrated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object that carries the argument is extrapolation, defined as the statistical operation of predicting the next value in a series, joined with the convex-hull picture of machine learning: a training set forms a minimal enclosure, and any output outside that enclosure is extrapolation while output inside it is interpolation. Wiener's extrapolation of stationary time series supplies the mathematical template—a message, like a missile trajectory, is a sequence of values whose future can be predicted within error bounds—and the paper transplants that template onto transformer-based next-token generation. The convex hull gives the argument its geometry: GPT's fake references are outputs that reach beyond the hull of factual training data while remaining statistically likely continuations of the prompt.
What would settle it
Take a known fabricated citation and locate its representation relative to the model's training set: if the invented reference falls inside the envelope of training examples, or is closer to memorized verbatim passages than to the boundary, then hallucination is interpolation rather than extrapolation and the paper's central reframing fails.
Extended reading notes
Core claim
The central discovery the paper is trying to establish is that artificial hallucination has another name: extrapolation. Each token an LLM emits is, in the paper's account, the next value in a time series; the model has learned statistical correlations between elements and projects a continuation beyond its training set. Under this description, a made-up citation is a perfectly sensible sequence—a plausible title, a likely author, a journal that would fit, a pseudo-DOI—rather than an error in the machine. The paper therefore claims that 'hallucination' pathologizes a functional behavior: the chatbot is not broken but extrapolating, and often extrapolating too well for the factual task it was asked to do.
Load-bearing premise
The load-bearing premise is that a chatbot's next-token generation really is a leap beyond its training data in the same sense that Wiener extrapolated a stationary time series, rather than a recombination of memorized or interpolated fragments.
Editorial extensions
If this is right
- Hallucinated references would be reclassified as design behavior rather than malfunction, so suppressing them means constraining a core capability, not repairing a defect.
- Extrapolation accounts for the same capacity that produces poems and novel prose and that lets chatbots avoid reproducing copyrighted text, giving the model a kind of intentional opacity.
- The interpolation/extrapolation distinction offers a geometric language for transparency: models could in principle report when an output lies outside the training hull.
- Evaluation would shift from asking whether every output matches a fact in the training set to asking how well the model extrapolates within acceptable error bounds.
- The historical lineage places LLM-era debates about trust in machines on a continuum with mid-century cybernetics, where predicting missiles and predicting messages were already the same mathematical problem.
Reading between the lines
- If the reframing holds, the practical safety question becomes 'how far beyond its training hull is a given output, and does the model signal that it has left the hull?' rather than 'how do we stop it from lying?'
- The convex-hull criterion suggests a quantitative test of the paper's claim: hallucinated references should sit near the boundary of the training-data distribution, whereas memorized or interpolated outputs should sit deep inside it.
- The copyright-evasion point implies extrapolation is also an economic design feature, not just an epistemic one, so transparency proposals will have to contend with the incentive to keep provenance deniable.
- One extension of the historical story would be to compare hallucination rates of interpolation-heavy systems, such as retrieval-augmented models, against pure next-token generators; a lower rate would support the claim that hallucination tracks extrapolation capacity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that large language models are best understood as machines of extrapolation, and that the phenomenon usually called "hallucination" is in fact an instance of the model extrapolating beyond its training data. The argument proceeds in two strands. The first is historical: the paper traces the concept of extrapolation through Norbert Wiener's wartime work on time series and missile prediction, connects it to Andrey Kolmogorov's parallel work, and reads the subsequent reprinting of Wiener's monograph as a Cold War public-relations maneuver. The second strand is conceptual: using the convex hull of a training set as a geometric criterion, the paper claims that outputs outside that hull are extrapolations, and that fabricated references produced by LLMs are such outputs. The paper concludes that hallucination is not a malfunction but evidence that the model is working as designed, potentially too well.
Significance. If the central claim were established, the paper would provide a provocative and genuinely useful reframing of AI hallucination, with consequences for how researchers, users, and regulators describe and evaluate LLM failures. The historical research on Wiener and Kolmogorov is a genuine contribution: it draws attention to underappreciated connections between wartime prediction theory and modern machine learning, and it does so with careful attention to archival detail and to the sociopolitical contexts of both scientists. The paper also productively connects contemporary LLM discussions to a longer intellectual history, which is valuable for interdisciplinary AI scholarship. However, the load-bearing conceptual claim—that hallucinated outputs are extrapolations—is not supported by any empirical or formal analysis. The definition of extrapolation via convex hulls is introduced but never operationalized on LLM outputs, and the paper does not engage with alternative mechanisms for hallucination such as memorization, interpolation, decoding errors, or sampling randomness.
major comments (3)
- [§3 and §4] The paper's central identification of hallucination with extrapolation is unsupported. In §3, extrapolation is defined as an output lying outside the convex hull of the training set, and interpolation as lying inside it. In §4, fabricated references are offered as examples of extrapolation. Yet the paper provides no evidence that a hallucinated citation actually lies outside the convex hull of the training data. At the token, embedding, or semantic level, a fabricated reference is typically a recombination of real author names, journal titles, and DOIs, which may well lie inside the convex hull of memorized exemplars. The definition in §3 is therefore never connected to the phenomenon in §4; the claim "artificial hallucination has another name: extrapolation" (p. 3) remains an assertion rather than a demonstrated equivalence.
- [§1 and §3] The analogy to Wiener's stationary time series extrapolation is metaphorical rather than substantive. Section 1 describes token generation as a time series and cites Wiener's theory, while §3 switches to a convex-hull definition that has no direct relationship to Wiener's linear stationary prediction theory. LLM token distributions are not stationary, transformers are not linear predictors, and the mean-square error framework of Wiener's work does not apply to next-token sampling in modern LLMs. The paper does not bridge this gap, so the historical appeal cannot bear the weight of the conceptual conclusion.
- [§1 and §4] The claim that a hallucinated output is "nothing more unreal than what was statistically probable in a series" (p. 2) dismisses without argument a substantial body of evidence that hallucination can arise from mechanisms other than extrapolation, such as memorization of training data, sampling randomness, decoding errors, or attention failures. Even if some hallucinations are extrapolative, the paper needs to argue that these are the typical or defining cases; otherwise, the reframing is a category error. The paper offers no such argument, and its dichotomy between "malfunction" and "extrapolation" is too coarse to capture the multi-causal nature of hallucination.
minor comments (5)
- [Throughout] There are numerous typographical and grammatical errors, e.g., "There could had been many paths" (p. 2), "recreats" (p. 8), "Guatarri" (p. 10), "Norvi" (works cited entry for Russell and Norvig), and "Marconian" (p. 23). These should be corrected in a revision.
- [Footnote 2] The footnote states that "interpolation and extrapolation are the same statistical method," but the main text in §3 clearly distinguishes them. This apparent contradiction should be resolved.
- [§2, historical citations] The paper cites Wiener's "A Scientist Rebels" as published in 1947 in the text and as 1946 in the works cited list; the access date and issue details should be reconciled.
- [§2] The claim that the 1949 reprinting of Wiener's monograph was a deliberate public-relations pivot away from the atomic bomb is presented as interpretation, but the evidence offered (the reprint itself, the preface) does not clearly support this causal claim; the claim should be labeled as speculative or supported with additional archival evidence.
- [§3] The discussion of Balestriero et al. (2021) and Cao and Yousefzadeh (2023) would benefit from a clearer explanation of the difference between "always amounts to extrapolation" and the author's more restricted claim, especially since the latter is the author's own prior work and is not summarized in enough detail for a reader to evaluate the distinction.
Circularity Check
No significant circularity: the historical and conceptual arguments stand independently; the one self-citation is non-load-bearing.
full rationale
The paper is a conceptual and historical argument, not a formal derivation. It contains no fitted parameters and no equations whose outputs are forced by construction. The central claim that LLMs extrapolate is anchored in external sources: Balestriero et al. (2021) for high-dimensional extrapolation, Wiener's 1942 treatise for the time-series notion, and the machine-learning literature on convex hulls. The only self-citation, Cao and Yousefzadeh (2023), appears in Section 3 as a caveat—'If we look at lower dimension models, not all amounts to extrapolation, but only specific types of predictions require it'—used to temper Balestriero's 'always' claim. It does not carry the paper's central premise, which is that LLM outputs are extrapolations. The identification of hallucination with extrapolation relies on an empirical assumption that fabricated citations lie beyond the training set's convex hull, and that assumption is not demonstrated; however, an unsupported premise is a correctness risk, not circularity. The paper does not define hallucination in terms of extrapolation, does not fit a parameter and then re-predict it, and does not invoke a self-authored uniqueness theorem. The self-citation is transparent, externally published, and not load-bearing, so it does not raise the circularity score.
Assumptions & free parameters
assumptions (3)
- domain assumption LLM token generation is a time-series prediction problem of the same kind as Wiener's extrapolation.
- ad hoc to paper Hallucinated outputs are extrapolations beyond the training data, not other failure modes.
- domain assumption Wiener's and Kolmogorov's extrapolation work is the correct origin for modern machine learning extrapolation.
Cite this review
Pith. "Pith review of How Large Language Models (LLMs) Extrapolate: From Guided Missiles to Guided Prompts." pith.science (2026). https://pith.science/paper/YB4A6M7N
@misc{pith2026250110361,
author = {Pith},
title = {Pith review of: How Large Language Models (LLMs) Extrapolate: From Guided Missiles to Guided Prompts},
year = {2026},
howpublished = {\url{https://pith.science/paper/YB4A6M7N}},
note = {Machine review of arXiv:2501.10361}
}
read the original abstract
This paper argues that we should perceive LLMs as machines of extrapolation. Extrapolation is a statistical function for predicting the next value in a series. Extrapolation contributes to both GPT successes and controversies surrounding its hallucination. The term hallucination implies a malfunction, yet this paper contends that it in fact indicates the chatbot efficiency in extrapolation, albeit an excess of it. This article bears a historical dimension: it traces extrapolation to the nascent years of cybernetics. In 1941, when Norbert Wiener transitioned from missile science to communication engineering, the pivotal concept he adopted was none other than extrapolation. Soviet mathematician Andrey Kolmogorov, renowned for his compression logic that inspired OpenAI, had developed in 1939 another extrapolation project that Wiener later found rather like his own. This paper uncovers the connections between hot war science, Cold War cybernetics, and the contemporary debates on LLM performances.
Reference graph
Works this paper leans on
-
[2]
Extrapolation, A History From Missiles to Word Projectiles To understand what extrapolation meant in a more distant era, before “cybernetics” and “information” had become words of interest, it is important to sketch the concerns in an older discipline - that of statistics. In the mid to late twentieth century, a snowballing number of mathematicians turned...
work page 2019
-
[4]
GPT Hallucinations Extrapolation, especially those that might result in false references and fake court reports, leaves us with the impression that something must not be working.11 But what exactly is not working, and for whom? For those who did not believe that machines could generate human-like dialogues, the initial response to the marvelous chatbot GP...
arXiv 2022
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.