{"id":"579601d7-c74b-46f7-99c2-8e42cfacbf06","arxiv_id":"2501.07124","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"K2 Diamond is a fully open 65B-parameter LLM that reaches Llama 2 70B-level performance on standard benchmarks.","lead":"The LLM360 team trained K2 Diamond, a 65-billion-parameter language model, from scratch and released its code, data, training logs, and checkpoints. The model roughly matches Llama 2 70B on 21 benchmarks while using a smaller pretraining corpus.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Loss-spike rollbacks leave the released 'exact training data sequence' ambiguously related to the actual run, so a third party cannot verify the central reproducibility claim.","rationale":"The paper is a credible engineering report: the model exists, the benchmarks are broadly consistent with the stated claims, and the release of code, checkpoints, logs, galleries, and loss-spike artifacts is genuinely valuable. The reader's condition on artifact fidelity is directionally correct, but the more precise and more load-bearing gap is the rollback/data-sequence mapping. The paper's own Section 4.4 shows that training was not a single uninterrupted pass through 360 chunks, yet Section 3 says the released data sequence is 'exact.' Without a precise statement of how restarts affect chunk consumption, the central reproducibility promise is not testable. This is an internal inconsistency, not a disagreement with external consensus, and it can be resolved by an audit of the released logs and data chunks. The concern does not invalidate the model or its performance, so the verdict should remain CONDITIONAL, with the condition sharpened to include a demonstrable alignment between the released data sequence and the actual training trajectory.","tokens_in":50816,"tokens_out":7205,"duration_ms":72790,"concrete_test":"Using the released Weights & Biases logs (wandb.ai/llm360/K2-Diamond) and the LLM360/K2Datasets chunk boundaries, reconstruct the actual sequence of global steps and data-batch indices consumed in the main K2 Diamond run. Verify that after each documented loss-spike restart, the step counter and data iterator resume exactly at the start of the next released chunk, with no chunk skipped, truncated, or replayed, and that the SHA-256 hash of each uploaded checkpoint N matches a model trained from checkpoint N-1 on exactly the released chunk N. If the logs show a chunk consumed twice or skipped, the 'exact data sequence' claim is falsified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3 promises 'the exact sequence of training data, segmented into chunks corresponding to each checkpoint,' and the paper's central claim is full 360-degree reproducibility. However, Section 4.4 documents two malignant loss spikes: 'we restarted training from earlier checkpoints to mitigate these issues' and 'allowed training to proceed for a few steps after each spike.' The paper never specifies how the released 360-chunk sequence maps onto the data actually consumed after these rollbacks: were spike-containing chunks fully replayed, partially skipped, or re-ordered in the main run? Does the main checkpoint numbering (1..360) correspond to one sequential pass through the 360 released chunks, with the post-spike checkpoints in separate repositories? If the released sequence is the intended chunk order rather than the actually consumed order/length, then even with all checkpoints, code, and logs in hand, a reproducer cannot reconstruct K2 Diamond. The footnote that only 120 of 380 checkpoints are uploaded (footnote 4/10) is a known incompleteness, but this rollback/data-sequence ambiguity is a stronger and unacknowledged threat to the transparency claim. It is an internal-consistency gap, not a disagreement with an external benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the LLM360 K2 project and its first model, K2 Diamond, a 65B-parameter transformer pretrained from scratch on roughly 1.4T tokens. The authors claim that K2 Diamond surpasses LLaMA-65B and rivals Llama2-70B on a 21-benchmark suite while using fewer FLOPs and tokens, and that the project is fully reproducible under the LLM360 360-degree open-source framework: code, data mix, exact data sequence, checkpoints, logs, and incident records are released. The paper also documents the TxT360 data curation pipeline, pretraining infrastructure and parallelism choices, loss-spike incidents and rollbacks, post-training and safety fine-tuning, and a qualitative longitudinal capability analysis. The central contribution is the transparency claim: a 65B model trained from scratch with a complete audit trail of data, code, checkpoints, and logs.","tokens_in":51031,"tokens_out":3653,"duration_ms":39300,"significance":"If the reproducibility claim holds, K2 Diamond would be a valuable community resource: it is the largest fully open-source 65B-scale model described to date, it re-evaluates comparison baselines with a shared harness, and it documents training incidents (loss spikes, hardware failures) and energy/carbon overhead in unusual detail. The longitudinal evaluations across 120 checkpoints and the release of evaluation galleries are genuinely useful for studying capability acquisition. These strengths are substantial and should be credited. However, the paper's own footnotes and incident descriptions reveal gaps between the advertised 'exact data sequence' + 'all checkpoints' and what is actually released, and the headline FLOPs-efficiency claim excludes the extra 30 days spent on loss-spike recovery. Because the openness/reproducibility claim is the paper's central contribution, these issues are load-bearing rather than cosmetic.","major_comments":[{"comment":"The central reproducibility claim is undermined by an unspecified relation between the released data sequence and the data actually consumed after the two malignant loss spikes. Section 3 promises 'the exact sequence of training data, segmented into chunks corresponding to each checkpoint,' and Section 4.4 states that the authors 'restarted training from earlier checkpoints' and 'allowed training to proceed for a few steps after each spike.' The paper never states whether the released 360 chunks correspond to one sequential pass through the intended order, whether spike-containing chunks were replayed, partially skipped, or re-ordered, or how checkpoint numbering (1..360) maps to the post-rollback runs, which are said to reside in separate repositories. Without this mapping, a third party cannot reconstruct K2 Diamond from the released artifacts, so the exact-sequence claim needs either a precise specification of the rollback/replay policy or a restatement of what the released sequence actually represents.","section":"Sec. 3 and Sec. 4.4"},{"comment":"The checkpoint availability statements are internally inconsistent. Section 3 lists 'Model Checkpoints: 140 intermediate model checkpoints, evenly distributed across stage 1 (120 checkpoints) and stage 2 (20 checkpoints),' while Section 4.3 says 360 and 20 chunks produce 'a total of 380 K2 Diamond checkpoints,' and footnote 4 says only 120 of the 380 are shared. The reader cannot tell how many checkpoints exist, how many are accessible, or whether the 140 number is a typo. Given that the paper advertises 'all intermediate model checkpoints saved during training' as part of the transparency claim, this discrepancy and the acknowledged storage-limited upload need to be resolved and clearly stated.","section":"Sec. 3 vs. Sec. 4.3, footnote 4"},{"comment":"The headline claim of 'an approximately 35% reduction in FLOPs compared to Llama2-70B' and the statement that K2 Diamond 'requires fewer FLOPs' exclude the additional 30 days of compute spent handling loss spikes, which the paper itself reports as an extra 129.3 MWh on top of the 430.8 MWh base training. Since loss-spike rollbacks involve recomputation from earlier checkpoints, the total FLOPs consumed by the project are materially higher than the useful-training FLOPs implied by 1.4T tokens. The paper should either report the efficiency claim net of this overhead or clearly separate 'training FLOPs to the final checkpoint' from 'total FLOPs expended,' and should qualify the abstract accordingly.","section":"Abstract, Sec. 1, and Power Consumption section"},{"comment":"The benchmark comparison that supports the 'surpasses LLaMA-65B and rivals Llama2-70B' claim is reported without error bars, confidence intervals, or significance tests. For example, the overall average scores are 57.20 (K2 Diamond) versus 57.11 (Llama2-70B), a difference of 0.09 points, and individual benchmarks show both favorable and unfavorable gaps. Because the evaluation harness is deterministic given the exact prompts and decoding settings, the authors could provide multi-seed or resampling-based variance estimates, or at minimum state the number of runs and the stability of the differences. As written, the central empirical comparison rests on averages that may be within run-to-run noise.","section":"Sec. 7, Table 15"}],"minor_comments":[{"comment":"The CommonCrawl cut-off date is listed as '2024-30,' which is not a valid month-day value; this is likely a typo and should be corrected.","section":"Table 1"},{"comment":"The paper says the model is trained on '1.4 trillion tokens' in the abstract, but Section 4.4 describes a major stage of 1.4T tokens plus a long-context stage of 69.3B tokens. Clarify whether the 1.4T figure already includes the long-context stage or whether the total is approximately 1.47T.","section":"Sec. 4.4 and Abstract"},{"comment":"The 'emergent and disappearing abilities' analysis uses a 90%-correct cutoff in Bucket 6 and averages over only 20 checkpoints per bucket; the authors do acknowledge that these are preliminary observations, but the presentation would benefit from stating the number of questions examined and the instability of the per-question estimates, since many questions show near-zero frequency in most buckets.","section":"Sec. 8.3 and Table 18"}],"recommendation":"major_revision","confidential_remarks":"The paper is a technical report whose value depends on the credibility of its artifact release. The two issues I would most want the editor to weigh are: (1) the data-sequence/rollback mapping question, which is not an external-consensus dispute but an internal gap in the paper's own claims, and (2) the inconsistency between the advertised checkpoint counts and the actual uploads, which a reader can verify immediately. Both are fixable in revision. I would not reject on the basis of benchmark methodology alone, since re-evaluation of baselines with a shared harness is a genuine strength, but the headline efficiency and performance claims need the stated qualifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Candid take: the release is real and novel—first fully-open 65B-scale pretraining with dense checkpoints, a public data sequence, and longitudinal eval curves across 120 points. For researchers who study training dynamics or loss spikes, this artifact set is genuinely new; previous fully-open efforts stopped at 7–12B. The paper also handles evaluation honestly: it re-runs the baselines, reports the settings, and even discloses the extra 30 days and 129 MWh spent on loss-spike recovery. The 'rivals Llama2-70B' claim is reasonable on their numbers (57.20 vs 57.11 average), though a couple of benchmark margins are within noise.\n\nThe place I push back is the reproducibility claim. Section 3 promises the exact data sequence, chunked per checkpoint, as the basis for 360-degree openness. But Section 4.4 says that after two malignant loss spikes they restarted from earlier checkpoints and kept training 'for a few steps after each spike,' with those post-spike checkpoints deposited in separate repos. The paper never explains how the released 360-chunk sequence corresponds to the data actually consumed across the rollbacks. Were the spike chunks fully replayed, partially skipped, or re-ordered? Does the main checkpoint number track the intended sequence or the actual one? Without that mapping, a third party holding the code, data, and checkpoints still cannot reconstruct K2 Diamond. That is a load-bearing gap for the paper's central promise, and it is not addressed. The footnote that only 120 of 380 checkpoints are uploaded is a separate, disclosed incompleteness; the rollback ambiguity is stronger and unacknowledged.\n\nSmaller issues: evaluation lacks error bars or significance tests, which matters for the near-tie with Llama2-70B; and the '35% fewer FLOPs' headline ignores the extra 30 days of wall-clock recovery time, although the paper does disclose that cost later. Neither is fatal, but both need a sentence or two of qualification.\n\nThe citation pattern looks fine: plausible prior art on open LLM releases, appropriate earlier LLM360 papers, and no glaring omissions. The paper does not try to fit the benchmarks; the data mix and hyperparameters are fixed and released, so circularity is not a concern.\n\nBottom line: this is a technically serious engineering report and a significant artifact release. It deserves peer review. A referee should ask the authors to specify the rollback-to-data mapping, to state exactly what 'exact data sequence' means in the presence of rollbacks, and to add basic variance information to the benchmark table.","headline":"Genuinely valuable 65B open-release, but the exact-reproducibility claim is undercut by the paper's own rollback description and incomplete checkpoint uploads.","tokens_in":51670,"tokens_out":3241,"would_cite":true,"duration_ms":32601,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that K2 Diamond, a 65-billion-parameter model trained completely from scratch on 1.4 trillion tokens, surpasses LLaMA-65B and rivals Llama2-70B on standard benchmarks, and it releases the code, data sequence, logs, and…","keywords":["open-source large language model","65B parameters","LLM pretraining transparency","reproducibility","loss spikes","TxT360 dataset","long-context training","LLM360 K2"],"falsifier":"Compare the hash of each released checkpoint against the training logs and the evaluation gallery outputs, then re-run the reported evaluation on the released final checkpoint and check whether the scores in the paper's benchmark table reproduce; a mismatch, or an inability to trace the released data sequence through to the published checkpoints, would falsify the central transparency claim.","tokens_in":50600,"feed_emoji":"🔓","tokens_out":6363,"duration_ms":57156,"temperature":0.7,"pith_summary":"This paper tries to establish that a 65-billion-parameter LLM can be trained entirely from scratch and released with enough of its record—code, exact data sequence, logs, and intermediate checkpoints—that other researchers can study and reproduce the run, not just use the weights. Its first model, K2 Diamond, is reported to surpass LLaMA-65B and to be competitive with Llama2-70B on a 21-benchmark suite while training on fewer tokens and FLOPs. If the claim holds, it means frontier-scale training no longer has to be a black box: loss spikes, capability emergence, and data-curation choices can be examined at 65B scale. It also means a 65B-class base model is available under Apache 2.0 for downstream work. The paper's own footnote, however, notes that only 120 of the 380 saved checkpoints are currently uploaded, so the promised complete checkpoint record is not yet fully public.","feed_headline":"Open-source 65B model rivals Llama-2-70B on fewer tokens","feed_subtitle":"K2 Diamond opens its full training record — code, data order, logs, and checkpoints — so the run can be audited.","key_machinery":"The load-bearing object is the auditable training record, not any single architectural innovation. The architecture follows LLaMA-65B; the claimed novelty is that every component that usually stays private is published: the TxT360 data curation pipeline with rich per-document metadata, the exact data mix and order (47% RefinedWeb, code via StarCoder, math boosted to 40% in stage 2, repetition and truncation rules), a two-stage curriculum, a 4D-parallelism setup (8-way tensor, 4-way pipeline, 15-way data parallel on 480 A100s) with a 2,040-token global batch, and a checkpoint after each data chunk (380 saved, 140 currently shared), plus training logs and incident reports on the two malignant loss spikes. This record is what would let a reader verify the training-from-scratch claim and study training dynamics.","core_discovery":"K2 Diamond is a 65-billion-parameter dense Transformer (80 layers, hidden size 8,192, no grouped-query attention) trained from scratch on 1.4 trillion tokens in two stages: a major stage at 2,048-token context and a long-context stage at 8,192 tokens using RoPE theta scaling. The paper reports an average score of 57.20 across 21 benchmarks, against 53.77 for LLaMA-65B and 57.11 for Llama2-70B, with its largest margins on code (HumanEval pass@1 of 32.0 versus 22.8 and 30.0) and medical QA, while using about 35% fewer FLOPs than Llama2-70B. It claims this makes K2 Diamond the first fully open-source LLM at this scale, releasing code, the exact ordering of training data, training logs, evaluation galleries, and checkpoints from during training.","pith_inferences":["If the released artifacts are exact, this makes a frontier-scale 'training from scratch' claim auditable for the first time, and it lets resource-constrained groups study scaling behavior without paying the full compute cost.","The paper's own checkpoint footnote suggests a practical tension in full-transparency releases: at more than 100GB per checkpoint, storage costs cap what can actually be shared, so complete openness at this scale may require new formats, incremental weight formats, or long-term archival commitments.","A natural testable extension is to use K2's released data sequence to pretrain a smaller model and check whether the same loss-spike and capability-acquisition patterns reproduce, which would separate model-scale effects from data-order effects.","The reported disappearance of some early abilities (e.g., certain GSM8K questions answered correctly at intermediate checkpoints but not at the end) invites a concrete follow-up: determine whether this reflects data-order interference, evaluation noise, or genuine forgetting during continued pretraining."],"forward_implications":["At 65B parameters, a fully open-source LLM can reportedly reach performance comparable to a leading open-weight recipe: the paper's 21-benchmark average is 57.20 for K2 Diamond, versus 53.77 for LLaMA-65B and 57.11 for Llama2-70B.","A two-stage curriculum on 1.4 trillion tokens is reported as sufficient to rival a model trained on 2 trillion tokens, giving evidence about data efficiency at the 65B scale.","Public loss-spike checkpoints and incident logs allow researchers to study why some spikes are benign and others destructive, using data rather than anecdote.","The longitudinal evaluations across 120 checkpoints give a fine-grained record of when reasoning, coding, medical, and bias-related behaviors appear or disappear during pretraining.","The Apache 2.0 license and released fine-tuning recipes make K2 a practical base for distillation, function calling, and domain adaptation without use restrictions."],"supporting_citations":[{"why":"Supplies the LLaMA-65B architecture and training setup that K2 Diamond follows and the primary model it claims to surpass.","marker":"Touvron et al., 2023a"},{"why":"Supplies the Llama2-70B baseline that K2 Diamond claims to rival on standard benchmarks with fewer FLOPs and tokens.","marker":"Touvron et al., 2023b"},{"why":"Defines the 360-degree open-source release principles that the K2 project scales to 65B parameters.","marker":"Liu et al., 2023d"},{"why":"Provides RefinedWeb, the largest single data source in K2 Diamond's pretraining mix at 47%.","marker":"Penedo et al., 2023"},{"why":"Provides StarCoder as the code data source and the tokenizer's 18 code-related special tokens.","marker":"Li et al., 2023a"},{"why":"Supplies the lm-evaluation-harness used to produce most of the reported benchmark scores.","marker":"Gao et al., 2023"},{"why":"Supplies FlashAttention-2, which the paper credits for speeding up the 480-GPU training run.","marker":"Dao, 2023"},{"why":"Supplies Algebraic-Stack, one of the math datasets used to boost math capability in both training stages.","marker":"Azerbayev et al., 2023"}],"fun_headline_variants":["Open-source 65B rivals Llama2-70B with 35% fewer FLOPs","Fully transparent 65B LLM rivals Llama2-70B on fewer tokens","K2 Diamond: open 65B model rivals Llama2-70B, all data released","First fully open 65B LLM rivals Llama2-70B, with full logs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the released code, data sequence, logs, and checkpoints are exactly the ones that produced the reported K2 Diamond scores; the paper's own footnotes say 380 checkpoints were saved but only 120 of the stage-1 checkpoints are uploaded so far, so the complete record is not yet available.","fun_headline_variants_meta":{"raw":{"variants":["Open-source 65B rivals Llama2-70B with 35% fewer FLOPs","Fully transparent 65B LLM rivals Llama2-70B on fewer tokens","K2 Diamond: open 65B model rivals Llama2-70B, all data released","First fully open 65B LLM rivals Llama2-70B, with full logs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000959,"raw_usage":{"total_tokens":4128,"prompt_tokens":1030,"completion_tokens":3098,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":3010}},"tokens_in":646,"tokens_out":3098,"duration_ms":20953,"temperature":1.0,"reasoning_tokens":3010,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:48:54.440835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the hash of each released checkpoint against the training logs and the evaluation gallery outputs, then re-run the reported evaluation on the released final checkpoint and check whether the scores in the paper's benchmark table reproduce; a mismatch, or an inability to trace the released data sequence through to the published checkpoints, would falsify the central transparency claim.","supporting_citations":[],"review_version":1}