{"id":"fb43527a-b7ac-449d-8217-c94ad70db47e","arxiv_id":"2502.08145","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AxoNN combines 3D parallel matrix multiplication with data parallelism to reach 1.423 exaflop/s on 6,144 H100 GPUs, and reports one-pass catastrophic memorization at the 70B scale that a masked-loss technique suppresses.","lead":"This paper introduces AxoNN, an open-source parallel training framework that reports sustained bf16 throughput above one exaflop on the Frontier and Alps supercomputers for GPT-style models. It also reports that 70-billion-parameter models can memorize documents in a single training pass, and shows that a masked-loss method can suppress this memorization in models up to 405B parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Memorization threshold claim is confounded by pre-existing memorization: the 405B model already exceeds 10% Exact Match on control documents before any continued training, so treated-bucket gains are not cleanly attributable to the AxoNN runs.","rationale":"The reader's verdict identifies the 405B control-group contamination as the weakest assumption, and my stress-test converges on the same point. The paper's performance results--620.1 Pflop/s on Perlmutter, 1.381 Eflop/s on Frontier, 1.423 Eflop/s on Alps--are reported with a plausible methodology (analytical FLOP counts, 10-iteration timing, last-8 averaging, and per-platform kernel tuning), and I do not find a concrete internal inconsistency in those numbers. The memorization section, however, is a distinct and prominent part of the abstract and title-level claim: it asserts a scale threshold for one-pass memorization and a mitigation. That threshold rests on comparing post-training rates to a control group that is not matched to the treated buckets and is already non-trivially memorized at 405B. Because the figure does not difference out bucket-specific 0-epoch rates, the causal attribution to continued training is not established at the largest scale, and the single-trial, 200-document design makes the 5% one-pass claim fragile. This does not rise to a rejection of the paper, since the systems contribution stands and the memorization finding may be true; it does mean the conditional verdict is appropriate and the memorization claims need revision or additional baseline measurements.","tokens_in":21331,"tokens_out":5888,"duration_ms":53531,"concrete_test":"Re-run the memorization experiment for the 70B and 405B models with bucket-specific baselines: before any continued training, record Exact Match on each of the four 200-document buckets separately using the pretrained checkpoints. Then report the marginal memorization rate (post-training rate minus that exact bucket's 0-epoch rate) for each epoch condition. If the 70B one-pass marginal rate is not significantly above zero across repeated trials, or if the 405B marginal rate is smaller than the 70B marginal rate, the claim that catastrophic memorization begins at 70B scale is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim includes a scientific finding: \"catastrophic memorization behavior\" emerges at the 70B scale, with roughly 5% of documents memorized in a single pass, and the Goldfish loss prevents it. The load-bearing evidence for this threshold is Figure 10, which compares Exact Match rates after 1, 4, and 6 epochs of continued training against a \"0 Ep\" control. For the 70B models the control is near zero, but for the 405B model the control already shows more than 10% exact reproduction before the experiment begins (Section VIII-C). This directly undermines the size-threshold conclusion: the 405B post-training rates are partly, perhaps largely, pre-existing memorization from the original Llama 3.1 pretraining, and the paper never reports the 0-epoch baseline for the specific treated buckets. The experiment also uses only 200 documents per bucket, a single trial for the 405B model, and an exact-match metric on the last 50 tokens, so the 70B \"one-pass 5%\" figure has a wide confidence interval and may not support the word \"catastrophic.\" The Goldfish-loss result in Figure 11 is likewise weakened because the 405B control already contains memorized documents, so reducing post-training rates \"to control levels\" does not demonstrate that memorization was prevented. The performance claims are largely independent of this issue and are documented in detail, so this is a concern about the memorization half of the central claim rather than the systems half.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AxoNN, an open-source framework for training large language models at scale, based on a four-dimensional hybrid parallel strategy that combines data parallelism with a three-dimensional parallel matrix-multiplication algorithm. It reports three main performance optimizations: automated BLAS kernel tuning, overlap of non-blocking collectives with computation, and a communication-cost model for selecting near-optimal process-grid configurations. The authors report sustained bf16 throughput of 620.1 Petaflop/s on 4,096 NVIDIA A100 GPUs, 1.381 Exaflop/s on 32,768 AMD MI250X GCDs, and 1.423 Exaflop/s on 6,144 NVIDIA H100 GPUs, with weak-scaling data across Perlmutter, Frontier, and Alps. The paper also presents a continued-pretraining study on Llama-family models claiming that 'catastrophic memorization' emerges at the 70B scale, with roughly 5% of documents memorized in one pass, and that the Goldfish Loss prevents this behavior. The performance model is validated against all possible configurations on 32 and 64 GPUs, and the FLOP count is checked against Nsight Compute.","tokens_in":21650,"tokens_out":4903,"duration_ms":40990,"significance":"If the performance results hold, AxoNN would be a notable open-source systems contribution: it demonstrates exascale bf16 throughput for LLM training on both NVIDIA and AMD GPU platforms, extends the previously demonstrated scale of distributed training, and provides a validated performance model plus a portable implementation. These strengths are substantial: the code is open source, the analytical FLOP count is checked empirically, and the performance model ranks nine of the ten fastest observed configurations correctly. The memorization study is scientifically interesting because it targets scales (70B and 405B) rarely accessible in academic settings, but its current experimental design does not yet support the size-threshold claim as cleanly as the systems contribution. The paper is best judged as two coupled contributions; the systems half is close to acceptance, while the memorization half needs additional analysis and controls.","major_comments":[{"comment":"The claim that catastrophic memorization emerges at the 70B scale is not cleanly supported by the presented evidence. The 405B model already reproduces more than 10% of the control documents at the 0-epoch baseline, as the paper itself acknowledges, so the post-training rates for that model are partly pre-existing and the untreated control cannot serve as a clean counterfactual for the treated buckets. The paper does not report the 0-epoch baseline for the specific treated buckets, and the 70B 'one-pass 5%' figure is based on 200 documents and three trials, corresponding to roughly 10 documents. Please report the 0-epoch baseline for each treated bucket, add confidence intervals or per-document variability for the exact-match rates, and either restrict the threshold claim to a formal comparison between the 70B and smaller models or soften the conclusion accordingly.","section":"Section VIII-C, Figure 10"},{"comment":"The Goldfish-loss conclusion is weakened by the same pre-existing memorization issue. In Figure 11, the 405B Goldfish curve is compared against the 0-epoch control, but that control already contains memorized documents; reducing the post-training rate 'to control levels' therefore does not establish that new memorization was prevented, only that the rate after training is no higher than the pre-existing rate. To support the prevention claim, the authors should show the matched 0-epoch baseline for the same buckets that are later trained with Goldfish, and ideally report the per-document overlap between documents memorized before and after training.","section":"Section VIII-D, Figure 11"},{"comment":"The headline sustained flop/s numbers are based on a single run per configuration, averaged over the last 8 of 10 iterations, and Section VI-B acknowledges significant run-to-run variability due to network congestion and file-system degradation. For claims of 'sustained' exascale throughput and for the comparison in Table I, at least a small number of repeated runs (or, minimally, per-iteration min/max values) should be reported for the 4,096-GPU Perlmutter, 32,768-GCD Frontier, and 6,144-GPU Alps configurations. Without this, the 620.1 Petaflop/s, 1.381 Exaflop/s, and 1.423 Exaflop/s figures do not have a stated uncertainty.","section":"Section VI-C and VII-B"}],"minor_comments":[{"comment":"In the final paragraph of Section VII-C, 'rain these models' should read 'train these models'.","section":"Section VII-C"},{"comment":"The verification that Narayanan et al.'s analytical FLOP count matches Nsight Compute is described only as a 'small experiment'; please provide the experimental details or a reference so that the verification can be reproduced.","section":"Section VI-C"},{"comment":"The empirical peak of 813 Tflop/s for the GH200 is attributed to an NVIDIA benchmark guide without a citation; please add a reference or measurement details.","section":"Section VI-C"},{"comment":"The decoding procedure for the memorization test is unspecified; please state whether generation is greedy or sampled, how many tokens are generated, and how truncation is handled, since the exact-match metric is sensitive to these choices.","section":"Section VIII-B"},{"comment":"References [13] and [27] are duplicates of the same ZeRO++ paper; remove one of them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The systems contribution is strong and likely within the journal's scope, but the memorization half is currently presented as a headline result and needs matched baselines, confidence intervals, and a clearer separation between pre-existing and newly acquired memorization. These issues are addressable with additional analysis and limited additional experiments, so I would not reject the paper on their account."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, unless the measurements are quietly wrong, AxoNN has the best-documented large-scale LLM training numbers I've seen from an open-source framework: 620 Pflop/s on 4,096 A100s, 1.381 Exaflop/s on 32,768 MI250X GCDs, 1.423 Exaflop/s on 6,144 H100s. The paper reports both advertised-peak and empirical-peak efficiencies, checks the FLOP formula against Nsight Compute, and validates its configuration model against exhaustive runs on 32 GPUs (9 of the top-10 predicted configs match the observed fastest). That is real, reproducible work, and the authors describe the measurement caveats—10-iteration runs, 8-iteration averages, acknowledged variability—better than most systems papers do.\n\nSecond, the memorization half is softer than the abstract suggests. The stress-test concern is right: the 405B model already reproduces >10% of the control documents before any continued training (Figure 10, 0 Ep bars), so the post-training rates are partly pre-existing from the original pretraining. The 70B models have near-zero controls, so the “5% in one pass” figure at 70B is not confounded, but the size-threshold story is weakened because the largest model's baseline is already high and its continued-training rate is actually slower than the 70B's. The 200-document buckets, single 405B trial, and last-50-token exact-match metric make “catastrophic” a strong word for the evidence. The Goldfish loss result is consistent with prior work but does not cleanly demonstrate prevention when the control levels are already elevated.\n\nOn the algorithm: the 4D scheme reduces to FSDP/ZeRO, Megatron, and ZeRO++, which the authors state honestly. The novelty is scale and integration, not the algorithm architecture. The performance claims also need code/data release to be fully checked; no artifacts are linked here.\n\nBottom line: this deserves a serious referee for the systems contribution. The memorization section should be revised to report 0-epoch baselines for the treated buckets, add repeated trials for the 405B, and soften the threshold language. I'd send it to peer review, not desk reject.\n\nWho's it for: anyone working on distributed training frameworks, and people interested in memorization measurement methodology. I'd cite the exascale numbers with caveats.","headline":"Systems half is strong and well-documented; the catastrophic-memorization threshold claim is confounded for the 405B and needs revision.","tokens_in":22247,"tokens_out":2942,"would_cite":true,"duration_ms":26613,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AxoNN sustains exascale bf16 throughput on three GPU supercomputers and shows that 70B-parameter models can memorize documents in a single pass.","keywords":["four-dimensional hybrid parallelism","3D parallel matrix multiplication","large language model training","exascale throughput","catastrophic memorization","Goldfish Loss","GPU supercomputing","weak scaling"],"falsifier":"Reproduce the 70B continued-training protocol with a held-out control bucket of 1,000 unseen documents and score memorization on 200-token spans; if the control bucket shows match rates as high as the trained buckets, or if longer-span matches vanish, the claimed single-pass catastrophic memorization would not be supported.","tokens_in":21134,"feed_emoji":"⚡","tokens_out":7314,"duration_ms":56696,"temperature":0.7,"pith_summary":"This paper presents AxoNN, an open-source framework for training GPT-style transformers on GPU supercomputers, and reports sustained half-precision (bf16) throughput of 620.1 Petaflop/s on 4,096 A100 GPUs, 1.381 Exaflop/s on 32,768 MI250X GCDs, and 1.423 Exaflop/s on 6,144 H100 GPUs. The speed comes from a four-dimensional hybrid parallel scheme that combines data parallelism with a three-dimensional parallel matrix multiplication algorithm, plus automated BLAS kernel tuning, aggressive overlap of non-blocking collectives with computation, and a communication model that picks near-optimal GPU grid configurations. Using access to that scale, the paper also reports a memorization study: models at 70B parameters and above reproduce training documents verbatim after a single pass, with roughly 5% of documents memorized in one epoch, while 1B-13B models stay near zero. A masking technique called Goldfish Loss, which drops random tokens from the loss computation, brings exact-match rates back down to control levels. If these results hold, large-model training becomes accessible to academic and public researchers, and the same experiments expose a privacy and copyright risk that grows with model scale.","feed_headline":"Open-source training hits exascale; 70B models memorize in one pass","feed_subtitle":"A four-dimensional parallel algorithm sustains 1.4 exaflops, and a masked loss stops verbatim recall.","key_machinery":"The load-bearing object is the four-dimensional virtual GPU grid: GPUs are arranged as $G_{\\text{data}} \\times G_x \\times G_y \\times G_z$, where data parallelism runs across the $G_{\\text{data}}$ groups and each group runs a three-dimensional parallel matrix multiplication scheme (derived from a classical 3D PMM algorithm) with the weight matrix sharded along the Z axis to save memory and all-gather, reduce-scatter, and all-reduce collectives moving data. A communication model assigns bandwidths to each hierarchy level and ranks configurations, while kernel tuning and overlap of non-blocking collectives with computation reduce time per batch. For memorization, the central mechanism is Goldfish Loss, which randomly masks tokens so that the cross-entropy loss is not computed on them, making exact reproduction at inference time very unlikely for the masked tokens.","core_discovery":"The central discovery is that a four-dimensional hybrid parallel algorithm — data parallelism combined with a 3D parallel matrix multiplication over a $G_x \\times G_y \\times G_z$ grid — lets one open-source stack train GPT-style models with near-ideal weak scaling to thousands of GPUs and sustain high fractions of peak bf16 throughput on three systems: 620.1 Pflop/s on 4,096 A100 GPUs, 1.381 Eflop/s on 32,768 MI250X GCDs, and 1.423 Eflop/s on 6,144 H100 GPUs. The performance model predicts fast configurations by summing ring-collective times with bandwidths that depend on the process-group hierarchy, and the kernel-tuning and overlap optimizations close the remaining gap. The same scale enables a second finding: exact memorization of whole documents emerges sharply between the 13B and 70B parameter scales, with roughly 5% of documents memorized after one pass and up to 67% after six epochs in the 70B models, and Goldfish Loss masking suppresses this memorization to levels comparable to the control data.","pith_inferences":["Editorial extension: the paper's threshold claim (catastrophic memorization appears between 13B and 70B parameters) is based on a coarse size ladder; a sweep with intermediate sizes would locate the threshold and test whether it shifts with batch size, learning rate, or data domain.","Editorial extension: if single-pass memorization is real, deduplicated corpora remain vulnerable at scale, so mitigations like Goldfish masking may need to become a default part of training rather than an optional audit.","Editorial extension: the exascale figures were measured over short runs (ten iterations, averaged over the last eight); a longer sustained run on a dedicated partition would show whether the throughput holds over full training jobs."],"forward_implications":["Large-scale LLM pretraining and fine-tuning no longer requires proprietary stacks: a sequential training codebase can be parallelized with AxoNN and reach exascale throughput on government and academic supercomputers.","The paper's time-to-solution estimates project that an 80B model can be trained on 2 trillion tokens in about 25.5 days on 8,192 Frontier GCDs, and a 640B model in about 15 months.","Memorization risk is a scale effect, not a repetition effect alone: 70B models memorize documents in a single pass, so even perfectly deduplicated corpora can leak training data.","Goldfish Loss reduces exact-match rates of large models to control levels, providing a training-time mitigation for verbatim memorization.","AxoNN's four-dimensional algorithm generalizes existing parallel schemes: with certain grid axes it reduces to fully sharded data parallelism, ZeRO-style sharding, or Megatron-style tensor parallelism."],"supporting_citations":[{"why":"Supplies the three-dimensional parallel matrix multiplication scheme at the core of AxoNN's 4D algorithm.","marker":"[26]"},{"why":"Provides the analytical FLOP-count formulation used to compute sustained flop/s and the strong-scaling baseline methodology.","marker":"[6]"},{"why":"Supplies the tensor-parallel layer-splitting scheme that AxoNN generalizes and a comparison point for scaling results.","marker":"[14]"},{"why":"Establishes the sharded data parallelism baseline that the 4D algorithm subsumes and compares against.","marker":"[11]"},{"why":"Introduces the Goldfish Loss masking technique used in the memorization mitigation experiments.","marker":"[50]"},{"why":"Frames the memorization problem and provides the prior quantification of memorization across language models.","marker":"[44]"},{"why":"Provides the pretrained Llama 2 checkpoints used in the continued-training memorization runs.","marker":"[21]"},{"why":"Provides the Llama 3.1 checkpoints, including the 405B model, used in the memorization experiments.","marker":"[49]"}],"fun_headline_variants":["AxoNN: 4D parallel training sustains 1.4 exaflops on 3 supercomputers","Open-source LLM training breaks exascale, fixes one-pass memorization","Exascale LLM training open-sourced; Goldfish Loss kills verbatim recall","4D parallel algorithm: 1.4 exaflops, plus a fix for single-pass memorization","AxoNN trains 405B models at exascale and masks memorization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The memorization results rest on treating exact reproduction of the final 50 tokens of 200 Wikipedia articles as memorization caused by the continued-training runs, even though the 405B model already reproduced over 10% of the control documents before any continued training.","fun_headline_variants_meta":{"raw":{"variants":["AxoNN: 4D parallel training sustains 1.4 exaflops on 3 supercomputers","Open-source LLM training breaks exascale, fixes one-pass memorization","Exascale LLM training open-sourced; Goldfish Loss kills verbatim recall","4D parallel algorithm: 1.4 exaflops, plus a fix for single-pass memorization","AxoNN trains 405B models at exascale and masks memorization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1517,"prompt_tokens":1029,"completion_tokens":488,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":645,"tokens_out":488,"duration_ms":27610,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T10:15:52.661278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce the 70B continued-training protocol with a held-out control bucket of 1,000 unseen documents and score memorization on 200-token spans; if the control bucket shows match rates as high as the trained buckets, or if longer-span matches vanish, the claimed single-pass catastrophic memorization would not be supported.","supporting_citations":[{"cited_title":"A three-dimensional approach to parallel matrix multiplication,","cited_arxiv_id":null,"evidence_quote":"Supplies the three-dimensional parallel matrix multiplication scheme at the core of AxoNN's 4D algorithm."},{"cited_title":"Megatron-lm: Training multi-billion parameter language models using model parallelism,","cited_arxiv_id":null,"evidence_quote":"Supplies the tensor-parallel layer-splitting scheme that AxoNN generalizes and a comparison point for scaling results."},{"cited_title":"Zero: Memory optimizations toward training trillion parameter models,","cited_arxiv_id":null,"evidence_quote":"Establishes the sharded data parallelism baseline that the 4D algorithm subsumes and compares against."},{"cited_title":"Quantifying memorization across neural language models,","cited_arxiv_id":null,"evidence_quote":"Frames the memorization problem and provides the prior quantification of memorization across language models."},{"cited_title":"Llama 2: Open foundation and fine-tuned chat models,","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained Llama 2 checkpoints used in the continued-training memorization runs."},{"cited_title":"The llama 3 herd of models,","cited_arxiv_id":null,"evidence_quote":"Provides the Llama 3.1 checkpoints, including the 405B model, used in the memorization experiments."}],"review_version":1}