{"id":"c67f8047-33af-4a05-b038-28bbab72de79","arxiv_id":"2501.04040","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A broad survey of LLM methods and applications, plus an empirical section on how code-rich pretraining may influence chain-of-thought reasoning, the details of which are not visible in the supplied text.","lead":"This paper surveys large language model foundations, scaling laws, training data, architectures, and applications, and reports new experiments on chain-of-thought and program-of-thought prompting in Llama models. A generalist should read it for a structured map of the current LLM landscape and the unresolved question of where reasoning abilities come from.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal claim linking code in pretraining data to CoT/PoT emergence is unverifiable as presented: Section 5 is missing, and the described Llama-variant comparisons cannot isolate code proportion from model size, training scale, and benchmark contamination.","rationale":"I read the paper in good faith as a survey whose only original, falsifiable contribution is the claim that pretraining data composition, especially code, influences the emergence of CoT/PoT abilities. The paper states this claim in the abstract and in Sections 1.2, 1.3, and 5, and it refers to experiments on Llama-family models using GSM8k and gsm-hard. The most load-bearing condition for that claim is causal identification: performance differences across Llama variants must be attributable to code proportion rather than to model size, training scale, architecture version, alignment, tokenizer, or benchmark memorization. The supplied manuscript does not satisfy this condition. The actual Section 5 content is missing from the review copy, so no tables, prompt templates, model versions, or contamination checks are available. Even accepting the description at face value, comparing off-the-shelf Llama models cannot isolate code fraction because Llama 1, Llama 2, Llama 3, and CodeLlama differ on many dimensions simultaneously. I therefore agree with the reader's weakest_assumption about uncontrolled confounds and benchmark validity. I do not claim the hypothesis is false; I claim it is unsupported by the evidence as presented, and the described design cannot support it. The survey portions are broadly consistent with known literature and contain no independently verifiable artifacts. Given that the central empirical section is absent, the appropriate verdict remains UNVERDICTED, and my read does not change the reader's verdict. My concrete test is deliberately minimal: it uses two publicly available models that share architecture and scale and differ mainly in code exposure, and it adds a contamination check to address the memorization alternative. If that check fails, the paper's central claim would need substantially stronger evidence before it could be accepted.","tokens_in":46598,"tokens_out":4398,"duration_ms":42426,"concrete_test":"Run base (non-instruction-tuned) Llama-2-7B and CodeLlama-7B on identical GSM8k and gsm-hard samples with identical CoT and PoT prompts and fixed decoding (temperature=0, same max new tokens, same few-shot exemplars). These two models share architecture and size; CodeLlama is initialized from Llama 2 and trained on ~500B tokens, ~85% code, so code exposure is the dominant difference. If CodeLlama-7B does not beat Llama-2-7B on gsm-hard, the code-exposure hypothesis fails this controlled check. Run a contamination check on the same samples: compute longest common substring/n-gram overlap with Llama 2 training data or verify the items appear in no public pretraining corpus; report accuracy on the clean subset. This directly tests whether code proportion, rather than model version/size or memorization, drives the claimed CoT/PoT gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Sections 1.2, 1.3, and 5 announce the paper's central original claim: pretraining data composition, specifically code, influences the emergence of CoT/PoT reasoning, supported by 'empirical evidences' on Llama-family models using GSM8k and gsm-hard. For this claim to hold, observed accuracy differences must be attributable to code proportion. The described design - comparing off-the-shelf Llama variants in LMStudio - cannot support that attribution. Llama 1, Llama 2, Llama 3, and CodeLlama differ simultaneously in parameter count (7B-70B), training tokens (1T vs 1.8T vs 15T), architecture/hyperparameters (GQA, tokenizer, RoPE base, context length), and data mix (code share 4.5% in Llama 1, roughly 8% in Llama 2, 17% in Llama 3; CodeLlama is a Llama 2 fine-tune on ~85% code). None of these are controlled in the visible text, and the actual Section 5 tables, prompts, and numbers are absent from the supplied manuscript. Additionally, GSM8k accuracy is a weak instrument for 'reasoning' because GSM8k is public and widely reproduced on the web; without contamination analysis, higher scores could reflect memorization. The paper itself notes the size-dependence debate (Section 2.1 reports CoT gains mainly above 60B/100B, while Section 4.2.2 says size is not decisive), so cross-size comparisons are especially confounded. Thus the central claim is not merely unproven: the design, as specified, cannot distinguish code exposure from version, scale, tuning, or data contamination.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a broad survey of large language models: it reviews the historical development of LMs, scaling laws, prominent model families (BERT, T5, GPT, Llama, Gemma, Claude), domain-specific applications, pre-training data and preprocessing, adaptation via instruction and alignment tuning, transformer architectures, and usage techniques such as in-context learning, chain-of-thought prompting, program-of-thoughts, planning, and retrieval-augmented generation. The abstract and Section 1 present the paper's advertised original contribution as an empirical investigation of whether the proportion of code in pre-training data influences the emergence of Chain-of-Thought and Program-of-Thought abilities, with experiments on Llama-family models evaluated on GSM8k and gsm-hard. In the version of the manuscript made available for review, Sections 1.2, 1.3, and the table of contents describe this study, but the body text of Section 5 itself is not present; the provided text ends midway through Section 3.5.4. The survey portions collate many well-known results and cite numerous primary sources, but they also contain several concrete errors, including an incorrect model size in Table 2 and garbled text in the tokenizer description.","tokens_in":46968,"tokens_out":5344,"duration_ms":54040,"significance":"If the empirical claim about code in pre-training data being a causal or enabling factor for CoT/PoT emergence were established with adequate controls, it would be a useful contribution to the ongoing debate on what drives apparent emergent reasoning in LLMs, and it would complement earlier observations about code training and step-by-step reasoning. The survey parts of the paper could serve as a broad introductory reference, especially for readers wanting a single narrative covering scaling laws, model families, adaptation, and prompting strategies. The paper's strengths are its breadth, its many pointers to primary sources, and its generally clear organization; its weaknesses are the absence of the Section 5 evidence from the reviewed text, the confounding factors inherent in the described Llama-variant comparison, and a number of factual and typographical errors that currently reduce the reliability of the survey as a reference. There are no machine-checked proofs or reproducible code artifacts in the visible text, so the paper must be judged on the quality of its synthesis and on the credibility of its empirical section.","major_comments":[{"comment":"The paper's central original claim—that the presence of code in pre-training data influences the emergence of CoT and PoT abilities—is announced in the abstract and in Section 1.3, but the actual content of Section 5 (subsections 5.1–5.4) is not present in the manuscript text supplied for review. Only the table of contents entries and the introductory description in Section 1.3 are visible. As reviewed, the claim is therefore unsupported: the reader cannot check the experimental setup, the prompts, the result tables, or the quoted generated text. If the full Section 5 exists in the authors' version, it must be included in the review copy; if it does not, the manuscript currently promises an empirical contribution it does not deliver.","section":"Abstract; Sections 1.2, 1.3, 5"},{"comment":"The experimental design described in the paper cannot isolate the proportion of code in pre-training data as the cause of CoT/PoT performance differences. Comparing off-the-shelf Llama variants, as described in Section 1.3, means that model size, training-token count, architecture details, tokenizer, context length, and alignment status all vary together with code proportion: Table 4 reports only 4.5% code for Llama 1, while the text reports roughly 8% for Llama 2, 17% for Llama 3, and about 85% code for CodeLlama, but these models also differ in parameters (7B–70B), training tokens (1T–15T), and other hyperparameters visible in Table 2. Without matched-size models, or explicit controls for scale and training budget, any observed accuracy differences on GSM8k or gsm-hard cannot be attributed to code exposure. The authors need to either provide such controls or substantially weaken the causal claim to a correlational one.","section":"Section 1.3; Section 5.2; Tables 2 and 4"},{"comment":"The manuscript is internally inconsistent about the role of model size in CoT ability, and this inconsistency is load-bearing for the Section 5 comparison. Section 2.1 reports that CoT prompting yields gains mainly when model size exceeds 60B and that advantages become more pronounced above 100B, while Section 4.2.2 is described as saying that size is not a deciding factor. Since the Llama-family comparison described in Section 1.3 crosses exactly the size range where this debate matters (7B–70B), the empirical section must explicitly address how scale is controlled or otherwise accounted for. As it stands, the reader cannot tell whether a performance gap is due to code proportion, model size, or both.","section":"Section 2.1 vs. Section 4.2.2"},{"comment":"Using accuracy on GSM8k and gsm-hard as the measure of reasoning ability requires a contamination analysis, because these datasets are public, widely reproduced, and frequently included in web scrapes used for pre-training. The paper's own Section 3.3.2 discusses the importance of deduplication and contamination, but the described Section 5 experiments do not mention any decontamination or holdout procedure. Higher scores on these benchmarks could in part reflect memorization rather than the emergence of step-by-step reasoning. The authors should report overlap analysis between the evaluation items and the pre-training corpora of the tested Llama variants, or use a fresh or adversarially filtered evaluation split.","section":"Section 5.2; GSM8k/gsm-hard"}],"minor_comments":[{"comment":"The table lists 'LLaMA 3.1 504B'; the correct size is 405B. The same table also appears to conflate Llama 3 and Llama 3.1 training-token counts; the caption should cite the specific source for each row.","section":"Table 2"},{"comment":"The heading 'Casual decoder' should read 'Causal decoder'; the same misspelling appears in the surrounding text and in Table 21's model-card entries. This is a terminology error in a central architectural taxonomy.","section":"Section 3.5.2 and Table 21"},{"comment":"The tokenizer description reads 'The result was cut from a tokenizer with 7 million tokens to only 2 17 tokens', which is garbled; it should presumably read '2^17 tokens' or a specific vocabulary size. Please correct the typesetting and state the final vocabulary size explicitly.","section":"Section 2.4.2 (BloombergGPT)"},{"comment":"There is an obvious typo 'GTP-4 GPT-4'; also 'GTP-3' appears earlier in the same section. Please run a spelling pass over model names.","section":"Section 2.3.3"},{"comment":"The model name 'Minstral' should be 'Mistral', and 'seetings' should be 'settings'. Several other typos appear throughout; the paper would benefit from professional copyediting.","section":"Section 2.3.5 and elsewhere"},{"comment":"The text says 'Within the four FinLLMs in Figure 8', but the figure containing BloombergGPT, FinMA, InvestLM, and FinGPT is Table 8, not Figure 8.","section":"Section 2.4.2"},{"comment":"The abstract defines 'PoT' as 'Plan of Thought', while Section 5.2 uses 'Program of Thought'. These should be reconciled, and the standard term 'Program of Thoughts' should be used consistently.","section":"Abstract and Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is the main argument for the paper's novelty, but the Section 5 evidence is not present in the review copy, and the experiment description that is present cannot support a causal reading. The survey itself is largely derivative of existing surveys, especially Zhao et al. [364], and contains several factual errors that need correction before it can serve as a reliable reference. I would ask the authors to supply the full Section 5 with results and controls, and to consider whether the paper is better framed as a survey with a modest empirical observation rather than as a study establishing a causal link between code proportion and CoT emergence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a broad survey of LLMs that advertises one original empirical contribution (Section 5: how code in pretraining data influences the emergence of chain-of-thought and program-of-thought reasoning). That section is not in the supplied manuscript, so the paper as reviewed is a restatement of known material with no verifiable new result. If the actual experiments are as described—comparing off-the-shelf Llama variants in LMStudio—the claim would not survive scrutiny, because Llama 1, 2, 3, and CodeLlama differ in size, training tokens, architecture, tokenizer, alignment, and code share all at once.\n\nWhat the paper does well: it is a competent compilation. The survey covers model families, scaling laws, pre-training data, tokenization, instruction tuning, RAG, CoT, and planning in a logical order, and it cites most of the relevant literature. Someone new to LLMs could use it as a map. The idea that code in the training mix might enable step-by-step reasoning is a real research question, and the authors frame it clearly.\n\nThe soft spots are not subtle. First, the visible text contains concrete errors: Table 2 lists 'LLaMA 3.1 504B' (no such model; the largest is 405B), the header 'Casual decoder' appears in Section 3.5, and the BloombergGPT tokenizer description ends with 'only 2 17 tokens' (missing an exponent). These are minor individually but suggest careless copyediting. Second, the paper contradicts itself on whether model size is decisive for CoT: Section 2.1 leans on the >60B threshold, Section 4.2.2 says size is not decisive, and the empirical section never reconciles this. Third, and most important, the central empirical claim is unverifiable as supplied. Even the described protocol—GSM8k/gsm-hard accuracy on off-the-shelf Llama checkpoints—cannot separate code proportion from scale, alignment, and benchmark contamination. The 'Plan of Thought' acronym is also wrong in the abstract: the literature calls it Program of Thought, which makes me wonder how carefully Section 5 was written.\n\nWho is this for? A student wanting a broad orientation, not a researcher needing accurate details. The survey portion could be made useful, but as it stands it does not deserve a serious referee without the actual experiments and a round of corrections. If the authors include Section 5 and tighten the controls, I would look again.\n\nMy recommendation: treat this as a desk reject with an invitation to resubmit a corrected version that includes the empirical details. It is not a paper that should be sent to referees in its current form.","headline":"A broad but error-prone LLM survey whose only novel claim (code data aids CoT emergence) sits in a missing Section 5, so the paper as supplied is a restatement with no verifiable new result.","tokens_in":47454,"tokens_out":3874,"would_cite":false,"duration_ms":35811,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that code in pre-training data helps elicit chain-of-thought and program-of-thought reasoning in LLMs, and reports experiments on Llama-family models using grade-school math benchmarks.","keywords":["large language models","Llama","chain-of-thought prompting","program-of-thought prompting","pre-training data composition","code in pre-training data","emergent abilities","GSM8k"],"falsifier":"A controlled pre-training study that keeps architecture, tokenizer, compute, and non-code data identical, varies only the code fraction—for example 0%, 5%, and 15%—and finds no monotonic improvement in Chain-of-Thought or Program-of-Thought accuracy on GSM8k and gsm-hard would falsify the paper's central claim.","tokens_in":46408,"feed_emoji":"🧠","tokens_out":8138,"duration_ms":74719,"temperature":0.7,"pith_summary":"This paper is mostly a survey of large language models, but it also advances a specific empirical claim: the presence of code in pre-training data helps elicit Chain-of-Thought (writing intermediate reasoning steps) and Program-of-Thought (writing code to compute the answer) reasoning. On Llama-family models, tested on grade-school arithmetic word problems from GSM8k and a harder variant called gsm-hard, the authors report that models with more code exposure perform better when prompted to reason step by step or to write a program that computes the answer. If this holds, data composition is a direct lever on reasoning ability, not just a correlate of model scale. The survey around the claim maps how LLMs are built, adapted, and applied, which is the context that makes the data-composition question worth settling.","feed_headline":"Code-heavy pretraining may unlock step-by-step reasoning in LLMs","feed_subtitle":"A test on Llama models across grade-school math links code-heavy pretraining to better step-by-step reasoning.","key_machinery":"The load-bearing object is the pre-training data mixture, and within it the code fraction, treated as an independent variable that can be read off the documented training mixtures of Llama-family models. Chain-of-Thought prompting—asking the model to write its intermediate reasoning steps—and Program-of-Thought prompting—asking it to write a program whose execution yields the answer—are the probes that reveal whether the code in pre-training has left a trace. GSM8k and gsm-hard supply the grade-school math problems on which the probes are scored. The argument moves by comparing models whose corpora contain different reported code proportions—about 4.5% for the original Llama mix, roughly 8% for Llama 2, and about 17% for Llama 3—and checking whether step-by-step accuracy tracks that proportion.","core_discovery":"On the paper's own terms, the central discovery is that step-by-step reasoning in large language models is not just a matter of scale or prompting: it is shaped by what was in the pre-training corpus, with code singled out as the ingredient that makes Chain-of-Thought and Program-of-Thought abilities emerge or become easier to elicit. The supporting experiments use the public Llama family, whose documented training mixtures contain different amounts of code, and probe them on GSM8k and gsm-hard. Under Chain-of-Thought prompting the model is asked to produce intermediate natural-language reasoning steps; under Program-of-Thought prompting it is asked to write an executable program. The authors interpret the observed performance ordering as evidence that code-heavy pre-training is an enabling factor for these reasoning behaviors.","pith_inferences":["A decisive test would be a controlled pre-training run that fixes architecture, tokenizer, compute, and non-code data, varying only the code fraction; the paper's between-model comparison cannot fully separate code from model size and alignment.","If the causal story holds, adding code to a training mix could give smaller models part of the reasoning gain currently attributed to scale, which would make step-by-step behavior cheaper to obtain.","The same reasoning suggests a caution for evaluation: gains on GSM8k and gsm-hard should be checked against code-form or solution leakage in the corpus before being credited to reasoning ability."],"forward_implications":["Data curation becomes a direct lever on reasoning ability: increasing code in the corpus could improve Chain-of-Thought and Program-of-Thought performance without increasing model size.","General-purpose corpora may need to include code even when the intended applications are not programming, because code may teach the structured, compositional thinking that step-by-step prompting relies on.","The hypothesis is checkable on other openly documented model families: their reported code fractions should predict their relative GSM8k and gsm-hard performance under Chain-of-Thought and Program-of-Thought prompting.","The two probing styles matter: models with more code exposure may show their advantage most clearly when reasoning is delegated to executable programs rather than expressed in natural language."],"supporting_citations":[{"why":"Introduces chain-of-thought prompting and the conjecture that training on code may contribute to this reasoning ability, which the paper sets out to test.","marker":"[230]"},{"why":"Documents the original Llama pre-training data mixture, including about 4.5% code from GitHub, providing the first data point for the code-fraction comparison.","marker":"[330]"},{"why":"Reports Llama 2's corpus with roughly 8% code, giving the middle data point in the code-exposure ladder.","marker":"[329]"},{"why":"Details Code Llama, a code-specialized continuation of Llama 2, which supplies the code-heavy endpoint of the model comparison.","marker":"[384]"},{"why":"Reports Llama 3's corpus with about 17% code, the largest code fraction among the compared family and the high-code endpoint.","marker":"[389]"},{"why":"Shows that models trained on code can pass unit tests and generate programs, grounding the premise that code training produces structured, executable reasoning.","marker":"[108]"}],"fun_headline_variants":["Code-rich training data boosts LLM step-by-step reasoning","Llama tests: code-heavy pretraining aids chain-of-thought","Why code in pretraining matters for LLM reasoning","Code-heavy corpora make LLMs reason step-by-step","Survey insight: code enables LLM reasoning leaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on attributing the observed GSM8k and gsm-hard performance differences to the code proportion in pre-training data, while model size, alignment tuning, tokenizer, and benchmark familiarity are not controlled across the compared models.","fun_headline_variants_meta":{"raw":{"variants":["Code-rich training data boosts LLM step-by-step reasoning","Llama tests: code-heavy pretraining aids chain-of-thought","Why code in pretraining matters for LLM reasoning","Code-heavy corpora make LLMs reason step-by-step","Survey insight: code enables LLM reasoning leaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1372,"prompt_tokens":988,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":604,"tokens_out":384,"duration_ms":4382,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:46.425316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled pre-training study that keeps architecture, tokenizer, compute, and non-code data identical, varies only the code fraction—for example 0%, 5%, and 15%—and finds no monotonic improvement in Chain-of-Thought or Program-of-Thought accuracy on GSM8k and gsm-hard would falsify the paper's central claim.","supporting_citations":[],"review_version":1}