Pith. sign in

REVIEW 5 major objections 5 minor 45 references

GPT as a Monte Carlo Language Tree: A Probabilistic Perspective

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that GPT-style models can be flattened into next-token probability trees that converge to the training corpus's own frequency tree as model size grows, and that more than 87% of model output tokens are among the corpus…

desk verdict A vivid tree visualization of LLM training-data fit, but the headline numbers rest on a tokenizer mismatch that needs fixing before the quantitative story is trustworthy. read the letter →

arxiv 2501.07641 v2 pith:YD6FTEDS submitted 2025-01-13 cs.CL

Kun-Peng Ning , Jia-Yu Yao , Yu-Yang Liu , Mu-Nan Ning , Li Yuan This is my paper · ORCID
classification cs.CL
keywords LargeLanguageModelsMonteCarloTreeData-TreeGPT-Treeprobabilisticpattern-matchingtokenbiashallucinationchain-of-thoughtreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that any natural-language training corpus can be flattened into a Monte Carlo Language Tree, a rooted tree with tokens as nodes, conditional next-token frequencies as edge probabilities, and one unique path per token sequence. Any GPT-style model can be flattened into the same kind of tree from its next-token probabilities. The paper claims that models trained on the same corpus produce similar trees, that larger models sit closer to the corpus tree, and that more than 87% of the tokens a GPT model outputs are among the top five next tokens of the corpus tree. This is presented as evidence that LLM generation is probabilistic pattern-matching against the statistics of the training data rather than formal rule-based reasoning. The tree view then offers a uniform explanation for token bias, hallucination, and why chain-of-thought prompting helps.

What carries the argument

The central object is the Monte Carlo Language Tree: a rooted tree in which each node is a token, each edge is the conditional probability of the next token given the prefix path, and every token sequence corresponds to exactly one path. For a corpus, the Data-Tree edge probabilities are empirical frequencies; for a model, the GPT-Tree edge probabilities are the model's next-token outputs. The load-bearing identity is Theorem 1: under ideal assumptions (unbounded capacity, infinite data, global optimum), the maximum-likelihood conditional probability equals the empirical conditional frequency, $p_{\theta^*}(w|h) = N(h,w)/N(h)$. This identity makes the Data-Tree the natural limit target for a fitted language model, so comparing GPT-Tree and Data-Tree through MSE and Recall@5 gives a quantitative handle on how close a model is to the corpus statistics.

What would settle it

Rebuild the Data-Tree from the exact training pipeline of an open model whose data order, tokenizer, and document packing are known, then test whether Recall@5 stays above 87% on generated tokens and whether tree MSE decreases monotonically with model size. A recall drop well below 87%, or a non-monotonic MSE curve, would falsify the convergence claim. Alternatively, on a symbolic task where correct answers follow a rule but not the corpus's high-frequency continuations, a model that still answers correctly would be evidence against the pattern-matching interpretation.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the reasoning process of LLMs is more likely to be probabilistic pattern-matching than formal reasoning, because each model inference appears to find a context pattern with maximum probability from the Data-Tree. Formally, the paper defines the Data-Tree $\theta^*$ by counting conditional token frequencies in the training corpus and the GPT-Tree $\hat{\theta}$ from the model's predicted next-token distributions, then argues that training a language model is an attempt to approximate the Data-Tree, written $\hat{\theta} \to \theta^*$. The experimental evidence is that GPT-Neo 125M, 1.3B, 2.7B and GPT-J-6B, all trained on The Pile, have visually similar GPT-Trees, that their MSE against the Data-Tree decreases with parameter count, and that Recall@5 stays above 87%, meaning the model's most probable next token is inside the Data-Tree's top five more than 87% of the time. From this the paper concludes that LLMs are not doing formal symbol manipulation but are selecting high-probability paths in a learned language tree.

Load-bearing premise

The argument assumes that the Data-Tree reconstructed by splitting The Pile into 2048-token chunks, discarding fragments shorter than 200 characters, and tokenizing with the GPT-NeoX tokenizer matches the training distribution the GPT models actually saw. If the models' real tokenizer, document packing, or deduplication differs from this reconstruction, then the reported convergence and 87% recall compare the models against the wrong target.

Editorial extensions

If this is right

  • Model scaling can be read as tree convergence: as GPT models grow, their GPT-Trees should move monotonically closer to the Data-Tree, so tree distance becomes a measurable proxy for how well a model has absorbed its training corpus.
  • Token bias is explained as path rerouting: a rare or perturbed token behaves like a low-mass branch that sends the model onto a wrong but high-probability subtree, which is why small input changes can flip an answer.
  • Hallucination is co-occurrence following: when the corpus tree strongly pairs terms like Canada and Toronto, the model follows that frequent path even when the generation is factually wrong.
  • Chain-of-thought works by inserting intermediate tokens that connect the input's node to the output's leaf, bridging the low-probability gap that would otherwise make direct generation fail.
  • Recall@5 above 87% sets a quantitative baseline: most generated tokens are drawn from the training corpus's high-frequency continuations, leaving only a small share of output that departs from the data tree.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the convergence claim holds, tree distance to the training corpus could serve as a scaling-law diagnostic, and adversarial perturbations could be predicted in advance by locating rare prefixes whose next branches lead into high-frequency wrong subtrees.
  • Editorial extension: the 87% recall number is conditional on the reconstructed Data-Tree matching the actual training pipeline; with the true tokenizer and chunking, the figure could move, so the specific number should be read as a lower bound on the method rather than a property of the model alone.
  • Editorial extension: a natural testable consequence is that text judged 'novel' or hallucinated should correspond to paths with low Data-Tree probability, which would let the tree be used as a hallucination detector without fine-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes representing any text corpus as a Monte Carlo Language Tree (Data-Tree), with nodes as tokens, edges as empirical conditional next-token frequencies, and unique paths for token sequences. A GPT-like model is similarly flattened into a GPT-Tree, with edges given by the model's predicted conditional probabilities. The authors prove (Appendix A, Theorem 1) that under ideal conditions the maximum-likelihood optimum equals the empirical conditional frequency distribution. Experiments on The Pile with GPT-Neo-125M/1.3B/2.7B and GPT-J-6B are used to claim that GPT-Trees visually resemble the Data-Tree, that larger models converge to the Data-Tree (decreasing MSE, increasing Recall@5), and that more than 87% of GPT output tokens can be recalled from the Data-Tree. The paper interprets these findings as evidence that LLM reasoning is probabilistic pattern-matching rather than formal reasoning, and offers qualitative explanations for token bias, hallucination, and Chain-of-Thought behavior.

Significance. If the quantitative claims were fully supported, the Monte Carlo Language Tree would offer a clean, interpretable probe of the relationship between an autoregressive language model and its training distribution. The theoretical result is standard but clearly presented, and the visualization methodology is intuitive and potentially useful. However, the current evidence is not yet sufficient to support the central convergence claim: the metrics depend on unreported hyperparameters, the Data-Tree tokenizer appears to be mismatched with at least three of the four evaluated models, the hand-selected root set is inconsistently described, and no uncertainty quantification is provided. The paper's broader interpretive claim about probabilistic pattern-matching versus formal reasoning goes beyond what the experiments can distinguish.

major comments (5)
  1. [Appendix B.2 and Eqs. (12)-(13)] The Data-Tree is built once using 'the BPE-based tokenizer used in the GPT-neo-X series, GPT-j-6B, and GPT3 models.' In fact, the GPT-Neo checkpoints (125M/1.3B/2.7B) use the GPT-2 byte-level BPE vocabulary of size 50257, whereas GPT-J-6B uses the GPT-NeoX/20B BPE vocabulary of size 50400. Consequently, for the three GPT-Neo models, the GPT-Tree probabilities p-hat and the Data-Tree probabilities p-star are defined on different token spaces, so the MSE and Recall@5 in Eqs. (12)-(13) compare token objects that are not the same. The authors must rebuild the Data-Tree separately with each model's tokenizer, or otherwise justify tokenizer identity, before the convergence claim can be evaluated.
  2. [Appendix B.1, Eqs. (12)-(13)] The metrics depend on a tree depth T and a top-K count (the N K_T notation in Eqs. (12)-(13)), yet no values of T or K are reported anywhere in the paper or figures. Without these values, the quantitative claims that MSE decreases and that recall exceeds 87% are not reproducible. The authors should report T and K explicitly and, ideally, show sensitivity of the trends to both parameters.
  3. [Section 3.1 and Figure 4] The root set is described as 'one word from the letters A to Z' and the figure captions say '26 Words', but the list in Section 2 contains only 23 words (As, Because, Could, Do, Even, For, Given, However, If, Just, Keep, Let, Many, Now, Once, Perhaps, Quite, Rather, Since, The, Under, Very, Where). This discrepancy must be corrected. Because these are hand-selected sentence-initial words, the authors should either justify their representativeness or report results over a larger random root set. Additionally, Figure 4 reports no error bars or significance tests, so the claimed monotone decrease in MSE and increase in Recall@5 may reflect noise rather than a systematic trend.
  4. [Appendix B.2 and Section 3.1] The Data-Tree reconstruction assumes that GPT-Neo-X and GPT-J-6B replicate the GPT-3 preprocessing of 2048-token chunks with fragments under 200 characters discarded. No evidence is provided that the actual packing, ordering, or deduplication of the training data for these checkpoints matches this reconstruction. Since the Data-Tree is meant to be the target distribution that the models approximate, a mismatch in preprocessing would make the reported convergence a comparison against the wrong target. The authors should document or empirically verify the preprocessing pipeline, or rebuild the Data-Tree from the actual training sequences used by each model.
  5. [Sections 2-3.1 and Theorem 1] Theorem 1 shows that, under ideal conditions, the maximum-likelihood optimum equals the empirical conditional-frequency distribution. This is a standard consistency-style result and does not by itself imply that each model inference 'finds a context pattern with maximum probability from the Data-Tree' or that LLM reasoning is 'probabilistic pattern-matching rather than formal reasoning.' The observed similarity between GPT-Tree and Data-Tree is partly a consequence of the training objective itself, not independent evidence for the interpretive claim. To support the stronger claim, the paper would need a direct comparison against alternative reasoning mechanisms, rather than only agreement with the training distribution.
minor comments (5)
  1. [Throughout] There are several typos and inconsistencies: 'Monte Calo' in the contribution list in Section 1, 'Coresponding Author' in the footnote, and Figure 3 shows edge labels such as p=1.0 while the text reports values like 0.996 for the same edges.
  2. [Eq. (12)] The notation N K_T is used before T and K are defined, and the parenthetical definition in Appendix B.1 does not specify how the total is computed for a tree with depth T and top-K branching. Please define all symbols before use and clarify the counting procedure.
  3. [Section 3.3] The Chain-of-Thought discussion is purely qualitative and presents no experiments connecting CoT paths to the GPT-Tree. As written, the 'I/O gap' and 'path Z' explanation is an analogy rather than a demonstrated mechanism.
  4. [Section 2] The statement that 'each sequence has a unique path' in the Data-Tree is imprecise, since multiple sequences can share the same prefix and hence the same path up to a given depth. Please clarify that uniqueness applies to the path as a sequence of tokens, not to the tree traversal.
  5. [Figures 2-3] The figures do not state how many branches are retained per node or how the tree is pruned. This matters for interpreting statements about similarity concentrating on the left half of the tree; please specify the branching and pruning procedure in the caption or appendix.

Circularity Check

2 steps flagged · score 7.0 of 10

Central 'convergence to Data-Tree' claim reduces by construction to an in-sample fit of models to their own training distribution.

  1. self definitional [Section 2 (Building Data-Tree, Eq. 2) and Appendix A (Theorem 1, Eq. 11)]
    "The Data-Tree θ∗ satisfies the following condition: pθ∗(x) = f (tn|t1, t2, ..., tn−1)/f (t1, t2, ..., tn−1), ∀x ∼ D, ∀n, (2) ... Thus the optimal solution is: pθ∗(w|h) = N (h, w)/N (h) = ˆp(w|h) ■ (11)"

    The Data-Tree is defined to be the empirical conditional frequency of the training corpus. Appendix A proves that the maximum-likelihood optimum is exactly this same empirical conditional frequency. Therefore Section 3.1's 'GPT models gradually converge to the Data-Tree' is not a test against an independent target; it is a restatement that MLE training on a corpus yields that corpus's conditional frequencies. The claimed convergence p̂θ → θ* is the optimization objective's closed-form solution, so the 'prediction' is equivalent to its input by construction.

  2. fitted input called prediction [Section 3.1 (Figure 4, Eqs. 12-13) and Appendix B.2]
    "It can be observed that the MSE results decrease as the model size increases, the larger the model, the closer it is to the Data-Tree. On the other hand, from the recall@5 results in Figure 5, more than 87% GPT output tokens will be recalled by Data-Tree."

    The four models are trained on The Pile and the Data-Tree is built from the same corpus using the same partitioning ('splitting each document into 2048 size slices and discard data chunks smaller than 200'). MSE and Recall@5 therefore measure how well each model reproduces the conditional statistics of its own training set, which is exactly the objective of next-token maximum-likelihood training. The 87% recall and monotone MSE decrease are in-sample fit statistics presented as 'findings' about pattern-matching; the predicted quantity is statistically forced by the training setup rather than independently derived.

full rationale

The paper's central quantitative claim, p̂θ → θ*, is not an externally testable prediction because θ* is defined as the conditional frequency of the very corpus on which the models were trained. Appendix A's Theorem 1 shows that the MLE optimum equals that same conditional frequency, so the observed convergence in Figure 4 is a check that training approaches its own optimization target on selected contexts. The 87% recall and MSE curves are in-sample fit statistics, not independent evidence for the 'probabilistic pattern-matching' conclusion. There is no load-bearing self-citation chain: reference [34] is a same-author hallucination paper cited only as 'similar conclusions', and it does not support the core convergence claim. A separate validity concern, noted but not counted as circularity, is that Appendix B.2 assumes a single tokenizer for GPT-Neo, GPT-J, and GPT-3, whereas the evaluated checkpoints may use different BPE vocabularies; that would make the Data-Tree/GPT-Tree comparison ill-defined, but it is a correctness risk rather than a circularity. Overall, the derivation reduces by construction to 'models fitted on a corpus approximate that corpus's conditional frequencies,' which warrants a score of 7.

Assumptions & free parameters 2 free parameters · 3 assumptions · 2 invented entities

The central derivation is standard maximum likelihood estimation; the paper's own Appendix A proves that the optimum equals the empirical conditional frequency. The load-bearing assumptions are the idealized conditions of Theorem 1, the accuracy of the reconstructed training-data statistics, and the representativeness of the hand-chosen roots. No external, out-of-corpus benchmark is used, so the ledger is dominated by assumptions tied to the paper's own setup.

free parameters (2)
  • Tree depth T and top-K count = not reported
    Equations 12 and 13 define MSE and Recall@5 over a tree of depth T with Top-K flattening, but the paper never states the numerical values used in Figures 2-4. These choices control the size of the compared trees and directly affect the reported recall and MSE numbers.
  • Root token selection = 23 listed, described as 26
    The set of starting words is hand-chosen ('As, Because, Could, ...'), and the number of items listed (23) does not match the stated count (26). The reported convergence may depend on this particular selection.
assumptions (3)
  • ad hoc to paper Ideal conditions: unbounded model capacity, infinite training data with every context infinitely frequent, and global optimization convergence.
    Assumption 1 in Appendix A is required for Theorem 1 (p*(w|h)=N(h,w)/N(h)), but the real models have finite capacity and see finite data, so the theorem does not strictly apply to the experiments.
  • domain assumption The empirical conditional frequencies computed from 2048-token chunks of The Pile with the GPT-NeoX tokenizer are the correct target distribution θ* for the evaluated models.
    Section B.2 describes this construction, but the actual training data, preprocessing, and tokenizer version used by the GPT models may differ from the reconstruction, which would bias the measured alignment.
  • ad hoc to paper The hand-picked sentence-initial words (and unstated tree depth) are representative of language model behavior.
    Section 2 selects 'one word from the letters A to Z' with no justification that these 23 contexts represent the full range of contexts the model encounters; the qualitative and quantitative conclusions depend on this representativeness.
invented entities (2)
  • Data-Tree (Monte Carlo Language Tree for the corpus) independent evidence
    purpose: Represents a training corpus as a rooted tree of conditional token frequencies, used as the ground-truth target θ* against which GPT models are compared.
    The Data-Tree is defined directly from the corpus and can in principle be recomputed by anyone with The Pile; however, it is an internal definitional construct, not an independently observed physical entity.
  • GPT-Tree (Monte Carlo Language Tree for the model) independent evidence
    purpose: Represents a GPT model's next-token probability distributions over the same shallow tree structure, allowing direct comparison with the Data-Tree.
    The GPT-Tree is reproducible by querying the public models, but it is simply a projection of the model's output distribution onto the paper's selected tree skeleton.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GPT as a Monte Carlo Language Tree: A Probabilistic Perspective." pith.science (2026). https://pith.science/paper/YD6FTEDS

@misc{pith2026250107641,
  author       = {Pith},
  title        = {Pith review of: GPT as a Monte Carlo Language Tree: A Probabilistic Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YD6FTEDS}},
  note         = {Machine review of arXiv:2501.07641}
}
read the original abstract

Large Language Models (LLMs), such as GPT, are considered to learn the latent distributions within large-scale web-crawl datasets and accomplish natural language processing (NLP) tasks by predicting the next token. However, this mechanism of latent distribution modeling lacks quantitative understanding and analysis. In this paper, we propose a novel perspective that any language dataset can be represented by a Monte Carlo Language Tree (abbreviated as ``Data-Tree''), where each node denotes a token, each edge denotes a token transition probability, and each sequence has a unique path. Any GPT-like language model can also be flattened into another Monte Carlo Language Tree (abbreviated as ``GPT-Tree''). Our experiments show that different GPT models trained on the same dataset exhibit significant structural similarity in GPT-Tree visualization, and larger models converge more closely to the Data-Tree. More than 87\% GPT output tokens can be recalled by Data-Tree. These findings may confirm that the reasoning process of LLMs is more likely to be probabilistic pattern-matching rather than formal reasoning, as each model inference seems to find a context pattern with maximum probability from the Data-Tree. Furthermore, we provide deeper insights into issues such as hallucination, Chain-of-Thought (CoT) reasoning, and token bias in LLMs.

Figures

Figures reproduced from arXiv: 2501.07641 by the authors.

Figure 1
Figure 1. A novel perspective for analyzing the intelligent abilities of LLMs by representing both the language dataset and the GPT models as Monte Carlo Language Trees. achieve impressive compression rates on modalities other than text, and [11] advocates for viewing the intelligent abilities of LLMs through the lens of compression. LLM attempts to perform lossless compression on large-scale data by next-token prediction pre… view at source ↗
Figure 2
Figure 2. The tree visualization results of GPT-X series models and Data-Tree. Each row and column represents a different model (or dataset) and token, respectively. These models are trained on the same large-scale 800GB dataset, i.e., The Pile. Different GPT-like language models trained on the same dataset have very high similarity in GPT-Tree visualization, and gradually converge to the Data-Tree, es￾pecially on the If toke… view at source ↗
Figure 3
Figure 3. A closer look of visualization of GPT-Tree and Data-Tree on the “If” token. GPT-Tree exhibits extremely high similarity in token prediction probability with Data-Tree. where f(·) the frequency function, and f(tn|t1, t2, ..., tn−1) is the condition frequency of tn based on its precedents [t1, ..., tn−1]. Our theoretical analysis in the Appendix A shows that the θ ∗ calculated by Eq.2 is the optimal solution of Eq.1. … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Average MSE and Recall@5 of 26 words. The MSE measures the probability difference between the GPT-Tree and Data-Tree, the smaller the value, the more similar it is. The Recall@5 measures how many GPT output tokens will be recalled by Data-Tree. The results show that th…
Figure 5
Figure 5. Figure 5: Understanding token bias issues in LLMs. (Left): The mathematical dataset contains a large number of patterns, like “Calculate {%f} {op} {%f}.\n{%f}”. We sample the original question (“Calculate 123 + 256.”) from the mathematical dataset, the GPT model could answer “35…
Figure 6
Figure 6. Figure 6: Understanding Hallucination Phenomenon. (Left): Input “What is the capital of Canada?” into the model, and both GPT-neo-125M and GPT-neo-2.7B models answer “The capital is Toronto”, exhibiting factuality hallucination. (Right): The GPT model leans heavily on the freque…
Figure 7
Figure 7. Figure 7: Understanding the effectiveness of Chain-of-Thought (CoT) from GPT-Tree. (Left): For some complex issues, there is a significant gap between input X and output Y , making it difficult for the GPT model to directly derive Y from X. (Right): From the perspective of GPT-T…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 18 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anad- kat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Lossless data compression with neural networks

    Fabrice Bellard. Lossless data compression with neural networks. URL: https://bellard. org/nncp/nncp. pdf, 2019

    work page 2019

  3. [3]

    Graph of thoughts: Solving elaborate problems with large language mod- els

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Ny- czyk, et al. Graph of thoughts: Solving elaborate problems with large language mod- els. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682–17690, 2024

    work page 2024

  4. [4]

    Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow

    Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata , 58(2), 2021

    work page 2021

  5. [5]

    Gpt-neox-20b: An open-source autoregressive language model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Gold- ing, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745 , 2022

    arXiv 2022

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  7. [7]

    Adapting lan- guage models to compress contexts

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting lan- guage models to compress contexts. arXiv preprint arXiv:2305.14788 , 2023

    arXiv 2023

  8. [8]

    Language modeling is compression

    Gr´ egoire Del´ etang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Ge- newein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchi- son, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309.10668, 2023

    arXiv 2023

Show all 45 references
  1. [9]

    Mathematical capabilities of chatgpt

    Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. Mathematical capabilities of chatgpt. Advances in neural information processing systems , 36, 2024

    2024

  2. [10]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020

    2020 arXiv

  3. [11]

    Ranking llms by compression

    Peijia Guo, Ziguang Li, Haibo Hu, Chao Huang, Ming Li, and Rui Zhang. Ranking llms by compression. arXiv preprint arXiv:2406.14171 , 2024

    2024 arXiv

  4. [12]

    Lossless and near-lossless compres- sion for foundation models

    Moshik Hershcovitch, Leshem Choshen, Andrew Wood, Ilias Enmouri, Peter Chin, Swaminathan Sundararaman, and Danny Harnik. Lossless and near-lossless compres- sion for foundation models. arXiv preprint arXiv:2404.15198 , 2024. 11

    2024 arXiv

  5. [13]

    A survey on halluci- nation in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on halluci- nation in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Informat...

    2023

  6. [14]

    Compressing llms: The truth is rarely pure and never simple

    Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Compressing llms: The truth is rarely pure and never simple. arXiv preprint arXiv:2310.01382, 2023

    2023 arXiv

  7. [15]

    A peek into token bias: Large language models are not yet genuine reasoners

    Bowen Jiang, Yangxinyu Xie, Zhuoqun Hao, Xiaomeng Wang, Tanwi Mallick, Weijie J Su, Camillo J Taylor, and Dan Roth. A peek into token bias: Large language models are not yet genuine reasoners. arXiv preprint arXiv:2406.11050 , 2024

    2024 arXiv

  8. [16]

    Can large language models reason and plan? Annals of the New York Academy of Sciences , 1534(1):15–18, 2024

    Subbarao Kambhampati. Can large language models reason and plan? Annals of the New York Academy of Sciences , 1534(1):15–18, 2024

    2024

  9. [17]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118 , 2023

    2023 arXiv

  10. [18]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1–35, 2023

    2023

  11. [19]

    Large language model guided tree-of-thought

    Jieyi Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291, 2023

    2023 arXiv

  12. [20]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229 , 2024

    arXiv

  13. [21]

    Alphazip: Neural network- enhanced lossless text compression

    Swathi Shree Narashiman and Nitin Chandrachoodan. Alphazip: Neural network- enhanced lossless text compression. arXiv preprint arXiv:2409.15046 , 2024

    2024 arXiv

  14. [22]

    A comprehensive overview of large language models

    Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muham- mad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435 , 2023

    2023 arXiv

  15. [23]

    Train- ing language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27...

    2022

  16. [24]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  17. [25]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  18. [26]

    Analysing math- ematical reasoning abilities of neural models

    David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing math- ematical reasoning abilities of neural models. arXiv preprint arXiv:1904.01557 , 2019. 12 GPT as a Monte Carlo Language Tree: A Probabilistic Perspective

    1904 arXiv

  19. [27]

    Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems , 36, 2024

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems , 36, 2024

    2024

  20. [28]

    Large language models can be easily distracted by irrelevant context

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Sch¨ arli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210– 31227. PMLR, 2023

    2023

  21. [29]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  22. [30]

    Llmzip: Lossless text compression using large language models

    Chandra Shekhara Kaushik Valmeekam, Krishna Narayanan, Dileep Kalathil, Jean- Francois Chamberland, and Srinivas Shakkottai. Llmzip: Lossless text compression using large language models. arXiv preprint arXiv:2306.04050 , 2023

    2023 arXiv

  23. [31]

    Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021

    Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021

    2021

  24. [32]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  25. [33]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022

    2022

  26. [34]

    Llm lies: Hallucinations are not bugs, but features as adversarial examples

    Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu-Yang Liu, and Li Yuan. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469, 2023

    2023 arXiv

  27. [35]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems , 36, 2024

    2024

  28. [36]

    Explainability for large language models: A survey

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology , 15(2): 1–38, 2024

    2024

  29. [37]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 , 2023

    2023 arXiv

  30. [38]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36: 46595–46623, 2023. 13

    2023

  31. [39]

    A survey on model com- pression for large language models

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model com- pression for large language models. Transactions of the Association for Computational Linguistics, 12:1556–1577, 2024. 14 GPT as a Monte Carlo Language Tree: A Probabilistic Perspective Appendix A. ...

    2024

  32. [40]

    The model has unbounded capacity (no parametric restrictions)

  33. [41]

    The training dataset size approaches infinity, with each context h appearing infinitely often ( N (h) → ∞)

  34. [42]

    The optimization converges to the global optimum. Theorem 1 (GPT Models Gradually Converge to the Data-T ree) Under Assump- tion 1, the optimal conditional probability distribution pθ∗(w|h) learned by an autoregressive language model through maximum likelihood estimation (Eq.1...

  35. [43]

    https://pile.eleuther.ai/

  36. [44]

    https://huggingface.co/datasets/EleutherAI/pile

  37. [45]

    Data-Tree

    https://echarts.apache.org/examples/zh/index.html#chart-type-sankey 17 p = f (t2|t1)/f (t1) from all data chunks as the edge. Repeating this process, we can obtain the “Data-Tree” started with t1 flattened by the Pile dataset. 18

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.