{"id":"d1512c76-b346-423f-9aa2-f01e04005744","arxiv_id":"2501.07641","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GPT models trained on a corpus are shown to approximate a tree of empirical next-token probabilities derived from the same corpus, with alignment increasing with model size.","lead":"The paper proposes a tree representation of both a training corpus and a GPT model, and reports that larger GPT models align more closely with the corpus's empirical next-token frequencies, recalling over 87% of model-generated tokens. A smart generalist might read this for a simple probabilistic picture of how GPT behaves, and for a lens on hallucination, token bias, and chain-of-thought prompting.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data-Tree is defined with a single tokenizer, but the evaluated GPT-Neo models appear to use a different BPE vocabulary than GPT-J-6B; the 87% recall and p̂θ→θ* convergence may therefore be comparisons across mismatched token spaces.","rationale":"The paper's central claim is that autoregressive LMs approximate the empirical conditional distribution of their training data, and that this approximation improves with scale. The theoretical appendix is a standard maximum-likelihood consistency argument and does not by itself distinguish the paper's perspective from ordinary empirical risk minimization. The empirical weight is carried entirely by the Data-Tree comparison: the 87% recall figure and the monotone MSE curve are the advertised quantitative evidence. The weakest point is therefore the construction of the Data-Tree itself. If the Data-Tree is built with a tokenizer that differs from a given model's tokenizer, then the compared probability distributions are not over the same objects, and every derived metric is uninterpretable. The reader's weakest assumption already mentions tokenizer version and data preprocessing, so the concern is not new, but it is the most load-bearing: it is concrete, checkable, and would invalidate the quantitative headline if confirmed. My verdict remains CONDITIONAL as the reader recommended, because the issue is empirically settled rather than logically fatal to the qualitative framing: if the test shows all models share a tokenizer and the results survive model-matched rebuilding, the quantitative claims can be restored; if not, the paper should be rejected or reframed as a qualitative proposal without the 87% and convergence claims.","tokens_in":10920,"tokens_out":7921,"duration_ms":76948,"concrete_test":"Download the tokenizer_config.json and vocab files for EleutherAI/gpt-neo-125M, gpt-neo-1.3B, gpt-neo-2.7B, and gpt-j-6b, and verify whether all four use the same BPE tokenizer. Then rebuild the Data-Tree twice on The Pile: once tokenized with the GPT-NeoX/GPT-J tokenizer and once with the GPT-2/GPT-Neo tokenizer. Recompute Figure 4's MSE and Recall@5 for each model against only its matching Data-Tree, and recompute the '>87%' statistic. If the monotone convergence by model size disappears, or the recall drops substantially, the central quantitative claim is an artifact of comparing mismatched token spaces.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is that GPT-Trees converge to the Data-Tree (p̂θ→θ*, Figure 4, Section 3.1). The Data-Tree is constructed once: The Pile is tokenized with 'the BPE-based tokenizer used in the GPT-neo-X series, GPT-j-6B, and GPT3 models' (Appendix B.2), and all conditional frequencies N(h,w)/N(h) are computed on those tokens. But the four evaluated checkpoints do not clearly share one tokenizer: GPT-Neo-125M/1.3B/2.7B are documented with the GPT-2 byte-level BPE (vocab 50257), while GPT-J-6B is documented with the GPT-NeoX/20B BPE (vocab 50400). If that is the case, the same textual context h produces different token sequences in the GPT-Tree and the Data-Tree, so the distributions p̂θ and θ* live on different vocabularies. MSE and Recall@5, as defined in Equations (12)-(13), then compare probabilities of token objects that are not the same, and the monotone decrease in Figure 4 is not a well-defined measure of convergence. Even if the tokenizers were identical, the paper does not show that its 2048-token/200-character chunking reproduces the actual packing, ordering, and deduplication used to train these checkpoints. Unless the Data-Tree is rebuilt separately with each model's own tokenizer and actual preprocessing, the headline '87% recall' and 'GPT models gradually converge to the Data-Tree' are unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes representing any text corpus as a Monte Carlo Language Tree (Data-Tree), with nodes as tokens, edges as empirical conditional next-token frequencies, and unique paths for token sequences. A GPT-like model is similarly flattened into a GPT-Tree, with edges given by the model's predicted conditional probabilities. The authors prove (Appendix A, Theorem 1) that under ideal conditions the maximum-likelihood optimum equals the empirical conditional frequency distribution. Experiments on The Pile with GPT-Neo-125M/1.3B/2.7B and GPT-J-6B are used to claim that GPT-Trees visually resemble the Data-Tree, that larger models converge to the Data-Tree (decreasing MSE, increasing Recall@5), and that more than 87% of GPT output tokens can be recalled from the Data-Tree. The paper interprets these findings as evidence that LLM reasoning is probabilistic pattern-matching rather than formal reasoning, and offers qualitative explanations for token bias, hallucination, and Chain-of-Thought behavior.","tokens_in":11307,"tokens_out":4628,"duration_ms":44383,"significance":"If the quantitative claims were fully supported, the Monte Carlo Language Tree would offer a clean, interpretable probe of the relationship between an autoregressive language model and its training distribution. The theoretical result is standard but clearly presented, and the visualization methodology is intuitive and potentially useful. However, the current evidence is not yet sufficient to support the central convergence claim: the metrics depend on unreported hyperparameters, the Data-Tree tokenizer appears to be mismatched with at least three of the four evaluated models, the hand-selected root set is inconsistently described, and no uncertainty quantification is provided. The paper's broader interpretive claim about probabilistic pattern-matching versus formal reasoning goes beyond what the experiments can distinguish.","major_comments":[{"comment":"The Data-Tree is built once using 'the BPE-based tokenizer used in the GPT-neo-X series, GPT-j-6B, and GPT3 models.' In fact, the GPT-Neo checkpoints (125M/1.3B/2.7B) use the GPT-2 byte-level BPE vocabulary of size 50257, whereas GPT-J-6B uses the GPT-NeoX/20B BPE vocabulary of size 50400. Consequently, for the three GPT-Neo models, the GPT-Tree probabilities p-hat and the Data-Tree probabilities p-star are defined on different token spaces, so the MSE and Recall@5 in Eqs. (12)-(13) compare token objects that are not the same. The authors must rebuild the Data-Tree separately with each model's tokenizer, or otherwise justify tokenizer identity, before the convergence claim can be evaluated.","section":"Appendix B.2 and Eqs. (12)-(13)"},{"comment":"The metrics depend on a tree depth T and a top-K count (the N K_T notation in Eqs. (12)-(13)), yet no values of T or K are reported anywhere in the paper or figures. Without these values, the quantitative claims that MSE decreases and that recall exceeds 87% are not reproducible. The authors should report T and K explicitly and, ideally, show sensitivity of the trends to both parameters.","section":"Appendix B.1, Eqs. (12)-(13)"},{"comment":"The root set is described as 'one word from the letters A to Z' and the figure captions say '26 Words', but the list in Section 2 contains only 23 words (As, Because, Could, Do, Even, For, Given, However, If, Just, Keep, Let, Many, Now, Once, Perhaps, Quite, Rather, Since, The, Under, Very, Where). This discrepancy must be corrected. Because these are hand-selected sentence-initial words, the authors should either justify their representativeness or report results over a larger random root set. Additionally, Figure 4 reports no error bars or significance tests, so the claimed monotone decrease in MSE and increase in Recall@5 may reflect noise rather than a systematic trend.","section":"Section 3.1 and Figure 4"},{"comment":"The Data-Tree reconstruction assumes that GPT-Neo-X and GPT-J-6B replicate the GPT-3 preprocessing of 2048-token chunks with fragments under 200 characters discarded. No evidence is provided that the actual packing, ordering, or deduplication of the training data for these checkpoints matches this reconstruction. Since the Data-Tree is meant to be the target distribution that the models approximate, a mismatch in preprocessing would make the reported convergence a comparison against the wrong target. The authors should document or empirically verify the preprocessing pipeline, or rebuild the Data-Tree from the actual training sequences used by each model.","section":"Appendix B.2 and Section 3.1"},{"comment":"Theorem 1 shows that, under ideal conditions, the maximum-likelihood optimum equals the empirical conditional-frequency distribution. This is a standard consistency-style result and does not by itself imply that each model inference 'finds a context pattern with maximum probability from the Data-Tree' or that LLM reasoning is 'probabilistic pattern-matching rather than formal reasoning.' The observed similarity between GPT-Tree and Data-Tree is partly a consequence of the training objective itself, not independent evidence for the interpretive claim. To support the stronger claim, the paper would need a direct comparison against alternative reasoning mechanisms, rather than only agreement with the training distribution.","section":"Sections 2-3.1 and Theorem 1"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: 'Monte Calo' in the contribution list in Section 1, 'Coresponding Author' in the footnote, and Figure 3 shows edge labels such as p=1.0 while the text reports values like 0.996 for the same edges.","section":"Throughout"},{"comment":"The notation N K_T is used before T and K are defined, and the parenthetical definition in Appendix B.1 does not specify how the total is computed for a tree with depth T and top-K branching. Please define all symbols before use and clarify the counting procedure.","section":"Eq. (12)"},{"comment":"The Chain-of-Thought discussion is purely qualitative and presents no experiments connecting CoT paths to the GPT-Tree. As written, the 'I/O gap' and 'path Z' explanation is an analogy rather than a demonstrated mechanism.","section":"Section 3.3"},{"comment":"The statement that 'each sequence has a unique path' in the Data-Tree is imprecise, since multiple sequences can share the same prefix and hence the same path up to a given depth. Please clarify that uniqueness applies to the path as a sequence of tokens, not to the tree traversal.","section":"Section 2"},{"comment":"The figures do not state how many branches are retained per node or how the tree is pruned. This matters for interpreting statements about similarity concentrating on the left half of the tree; please specify the branching and pruning procedure in the caption or appendix.","section":"Figures 2-3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is potentially publishable, but the current quantitative evidence is not reproducible because of the unreported T and K parameters and the likely tokenizer mismatch between the Data-Tree and the GPT-Neo models. These are fixable with additional experiments and re-analysis, so I do not recommend rejection, but the revision must include per-model tokenizer-consistent Data-Trees, explicit metric hyperparameters, and uncertainty quantification. The interpretive claim about probabilistic pattern-matching versus formal reasoning should also be substantially softened unless directly tested."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you want a picture of GPT as a lookup table over its training data. The Data-Tree/GPT-Tree idea is a clean way to show that autoregressive LMs approximate the empirical next-token distribution of their corpus, and the 87% recall figure would be a genuinely useful measurement if it were computed correctly. The paper is also honest enough to put the standard MLE result in Appendix A, even though it dresses it as Theorem 1.\n\nWhat is actually new: the tree representation itself, the visualizations, and the attempt to quantify convergence with MSE and Recall@5. The qualitative sections on token bias and CoT are plausible and clearly explained, but they are interpretations, not results. I would not take the 'probabilistic pattern-matching rather than formal reasoning' conclusion from these experiments; the data only show that models resemble their training distribution, which we already knew from MLE.\n\nThe soft spots are real. The stress-test note about tokenizers checks out. GPT-Neo 125M/1.3B/2.7B use the GPT-2 byte-level BPE, while GPT-J-6B uses the GPT-NeoX tokenizer. Building the Data-Tree once with a single tokenizer means the GPT-Tree and Data-Tree are not on the same vocabulary for some of the models, so the MSE and Recall@5 numbers in Figure 4 and the 87% recall are not well-defined comparisons. The paper needs to rebuild the Data-Tree with each model's tokenizer, or restrict the comparison to a common token space.\n\nAlso, T and top-K in Equations 12-13 are never reported; the root set is described as 26 words but only 23 are listed; and there are no error bars or significance tests. These are fixable, but right now the quantitative claims rest on undisclosed choices. The circularity of deriving the target from the same Pile corpus is real, though not fatal; it means the interesting question is not whether bigger models get closer, but how fast and with what finite-corpus deviations.\n\nBottom line: this is worth a serious referee, but only after the tokenizer issue is resolved. Send it to review with a request for major revision. If the authors fix the comparison and report their hyperparameters, the 87% recall and the convergence trend would be a genuinely quotable result. As it stands, I would not cite it.","headline":"A vivid tree visualization of LLM training-data fit, but the headline numbers rest on a tokenizer mismatch that needs fixing before the quantitative story is trustworthy.","tokens_in":11819,"tokens_out":2968,"would_cite":false,"duration_ms":28710,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that GPT-style models can be flattened into next-token probability trees that converge to the training corpus's own frequency tree as model size grows, and that more than 87% of model output tokens are among the corpus…","keywords":["Large Language Models","Monte Carlo Language Tree","Data-Tree","GPT-Tree","probabilistic pattern-matching","token bias","hallucination","chain-of-thought reasoning"],"falsifier":"Rebuild the Data-Tree from the exact training pipeline of an open model whose data order, tokenizer, and document packing are known, then test whether Recall@5 stays above 87% on generated tokens and whether tree MSE decreases monotonically with model size. A recall drop well below 87%, or a non-monotonic MSE curve, would falsify the convergence claim. Alternatively, on a symbolic task where correct answers follow a rule but not the corpus's high-frequency continuations, a model that still answers correctly would be evidence against the pattern-matching interpretation.","tokens_in":10698,"feed_emoji":"🌳","tokens_out":8234,"duration_ms":67334,"temperature":0.7,"pith_summary":"The paper proposes that any natural-language training corpus can be flattened into a Monte Carlo Language Tree, a rooted tree with tokens as nodes, conditional next-token frequencies as edge probabilities, and one unique path per token sequence. Any GPT-style model can be flattened into the same kind of tree from its next-token probabilities. The paper claims that models trained on the same corpus produce similar trees, that larger models sit closer to the corpus tree, and that more than 87% of the tokens a GPT model outputs are among the top five next tokens of the corpus tree. This is presented as evidence that LLM generation is probabilistic pattern-matching against the statistics of the training data rather than formal rule-based reasoning. The tree view then offers a uniform explanation for token bias, hallucination, and why chain-of-thought prompting helps.","feed_headline":"Over 87% of GPT outputs match the corpus tree's top tokens","feed_subtitle":"Bigger models track the corpus tree more closely, backing a pattern-matching rather than formal-reasoning view of LLMs.","key_machinery":"The central object is the Monte Carlo Language Tree: a rooted tree in which each node is a token, each edge is the conditional probability of the next token given the prefix path, and every token sequence corresponds to exactly one path. For a corpus, the Data-Tree edge probabilities are empirical frequencies; for a model, the GPT-Tree edge probabilities are the model's next-token outputs. The load-bearing identity is Theorem 1: under ideal assumptions (unbounded capacity, infinite data, global optimum), the maximum-likelihood conditional probability equals the empirical conditional frequency, $p_{\\theta^*}(w|h) = N(h,w)/N(h)$. This identity makes the Data-Tree the natural limit target for a fitted language model, so comparing GPT-Tree and Data-Tree through MSE and Recall@5 gives a quantitative handle on how close a model is to the corpus statistics.","core_discovery":"On the paper's own terms, the central claim is that the reasoning process of LLMs is more likely to be probabilistic pattern-matching than formal reasoning, because each model inference appears to find a context pattern with maximum probability from the Data-Tree. Formally, the paper defines the Data-Tree $\\theta^*$ by counting conditional token frequencies in the training corpus and the GPT-Tree $\\hat{\\theta}$ from the model's predicted next-token distributions, then argues that training a language model is an attempt to approximate the Data-Tree, written $\\hat{\\theta} \\to \\theta^*$. The experimental evidence is that GPT-Neo 125M, 1.3B, 2.7B and GPT-J-6B, all trained on The Pile, have visually similar GPT-Trees, that their MSE against the Data-Tree decreases with parameter count, and that Recall@5 stays above 87%, meaning the model's most probable next token is inside the Data-Tree's top five more than 87% of the time. From this the paper concludes that LLMs are not doing formal symbol manipulation but are selecting high-probability paths in a learned language tree.","pith_inferences":["Editorial extension: if the convergence claim holds, tree distance to the training corpus could serve as a scaling-law diagnostic, and adversarial perturbations could be predicted in advance by locating rare prefixes whose next branches lead into high-frequency wrong subtrees.","Editorial extension: the 87% recall number is conditional on the reconstructed Data-Tree matching the actual training pipeline; with the true tokenizer and chunking, the figure could move, so the specific number should be read as a lower bound on the method rather than a property of the model alone.","Editorial extension: a natural testable consequence is that text judged 'novel' or hallucinated should correspond to paths with low Data-Tree probability, which would let the tree be used as a hallucination detector without fine-tuning."],"forward_implications":["Model scaling can be read as tree convergence: as GPT models grow, their GPT-Trees should move monotonically closer to the Data-Tree, so tree distance becomes a measurable proxy for how well a model has absorbed its training corpus.","Token bias is explained as path rerouting: a rare or perturbed token behaves like a low-mass branch that sends the model onto a wrong but high-probability subtree, which is why small input changes can flip an answer.","Hallucination is co-occurrence following: when the corpus tree strongly pairs terms like Canada and Toronto, the model follows that frequent path even when the generation is factually wrong.","Chain-of-thought works by inserting intermediate tokens that connect the input's node to the output's leaf, bridging the low-probability gap that would otherwise make direct generation fail.","Recall@5 above 87% sets a quantitative baseline: most generated tokens are drawn from the training corpus's high-frequency continuations, leaving only a small share of output that departs from the data tree."],"supporting_citations":[{"why":"Supplies The Pile, the 800GB corpus from which the Data-Tree is built via token-frequency counting.","marker":"[10]"},{"why":"Supplies the GPT-3 data-partitioning rules (2048-token chunks, discard fragments under 200 characters) used to reconstruct the Data-Tree.","marker":"[6]"},{"why":"Provides the GPT-Neo 125M/1.3B/2.7B models whose next-token probabilities form the GPT-Trees.","marker":"[4]"},{"why":"Provides GPT-J-6B, the largest model tested, completing the model-size trend toward the Data-Tree.","marker":"[31]"},{"why":"Documents the GPT-NeoX tokenizer used to tokenize the Data-Tree chunks.","marker":"[5]"},{"why":"Defines chain-of-thought prompting, the phenomenon the GPT-Tree path-filling account explains.","marker":"[33]"},{"why":"Introduces token bias in LLMs, the fragility the authors explain as rare tokens inducing wrong tree paths.","marker":"[15]"},{"why":"Supplies the mathematics QA dataset used to construct the token-bias perturbation experiments.","marker":"[26]"},{"why":"Discusses hallucination as co-occurrence bias, which the tree view reframes as following high-frequency corpus paths.","marker":"[34]"}],"fun_headline_variants":["Over 87% of GPT tokens recalled from the corpus tree","LLM reasoning looks like probabilistic pattern-matching, not logic","Bigger GPT models track the training data tree more closely","Data-Tree: LLMs pick high-probability context patterns","GPT's high recall from corpus tree backs pattern-matching over reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the Data-Tree reconstructed by splitting The Pile into 2048-token chunks, discarding fragments shorter than 200 characters, and tokenizing with the GPT-NeoX tokenizer matches the training distribution the GPT models actually saw. If the models' real tokenizer, document packing, or deduplication differs from this reconstruction, then the reported convergence and 87% recall compare the models against the wrong target.","fun_headline_variants_meta":{"raw":{"variants":["Over 87% of GPT tokens recalled from the corpus tree","LLM reasoning looks like probabilistic pattern-matching, not logic","Bigger GPT models track the training data tree more closely","Data-Tree: LLMs pick high-probability context patterns","GPT's high recall from corpus tree backs pattern-matching over reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000998,"raw_usage":{"total_tokens":4264,"prompt_tokens":1021,"completion_tokens":3243,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":3156}},"tokens_in":637,"tokens_out":3243,"duration_ms":21538,"temperature":1.0,"reasoning_tokens":3156,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:38:17.001681+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the Data-Tree from the exact training pipeline of an open model whose data order, tokenizer, and document packing are known, then test whether Recall@5 stays above 87% on generated tokens and whether tree MSE decreases monotonically with model size. A recall drop well below 87%, or a non-monotonic MSE curve, would falsify the convergence claim. Alternatively, on a symbolic task where correct answers follow a rule but not the corpus's high-frequency continuations, a model that still answers correctly would be evidence against the pattern-matching interpretation.","supporting_citations":[{"cited_title":"Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow","cited_arxiv_id":null,"evidence_quote":"Provides the GPT-Neo 125M/1.3B/2.7B models whose next-token probabilities form the GPT-Trees."}],"review_version":1}