Pith. sign in

REVIEW 4 major objections 6 minor 22 references

Supernova: Achieving More with Less in Transformer Architectures

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 650M-parameter transformer reaches 90% of a 1B model's performance with 35% fewer parameters and 100B training tokens.

desk verdict The 90%-of-1B-with-100B-tokens claim is undone by the paper's own training arithmetic; the tokenizer compression result is the only part I'd take seriously. read the letter →

arxiv 2507.15773 v2 pith:5RWRKINF submitted 2025-07-21 cs.CL cs.AIcs.LG

Andrei-Valentin Tanase , Elena Pelican This is my paper
classification cs.CLcs.AIcs.LG
keywords 650M-parametertransformerbyte-levelBPEGroupedQueryAttentionRoPEdataefficiencyzero-shotevaluationscalinglawstokenizercompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Supernova is a 650M-parameter decoder-only transformer built to test whether architectural choices and tokenizer quality can substitute for scale and data volume in the sub-billion parameter regime. The paper's central claim is that careful component selection—RoPE, GQA with a 3:1 query-to-key/value ratio, RMSNorm, SwiGLU—plus a custom English-optimized byte-level BPE tokenizer with a 128,000-token vocabulary lets the model reach about 90% of the average zero-shot performance of Llama 3.2 1B while using 35% fewer parameters and 100B training tokens. That training budget is one to two orders of magnitude below the data used by comparable open models, so the result, if it holds, would mean efficiency engineering can move the cost-performance frontier instead of raw scale. The paper also reports deployment-side gains: faster inference, lower memory, and lower cost per token.

What carries the argument

The mechanism that carries the argument is the pair of tokenizer and attention configuration. The tokenizer is a byte-level BPE trained on English with a 128,000-token vocabulary; the paper reports 4.78 characters per token on WikiText-103, about 7.7% better than Llama 3.2's tokenizer, so the model sees more semantic content in its fixed 2048-token window. The attention configuration is Grouped Query Attention with 4 key-value heads shared by 12 query heads, a 3:1 compression that reduces the KV cache from 48MB to 16MB per layer in the paper's accounting. RoPE, RMSNorm, and SwiGLU fill out the block, but they are standard modern components; the paper's case for synergy is that position enters only the Q/K vectors, so sharing V across query groups costs little, and the high-compression tokenizer amplifies every architectural gain.

What would settle it

Reproduce Table 3 under a single evaluation harness: run Supernova and Llama 3.2 1B with identical prompt templates, zero-shot settings, sample counts, and random seeds on the same ten tasks, then recompute the unweighted average. If the ratio is not close to 90.29%, the central claim fails; a quicker check is to retrain or re-run the tokenizer on WikiText-103 and confirm the 4.78 characters-per-token figure against the stated BPE merge procedure.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a 650M model trained on 100B tokens can close most of the measured gap to a 1B model. In Table 3, Supernova's unweighted zero-shot average across ten benchmarks is 43.09, versus 47.73 for Llama 3.2 1B, which the paper rounds to 90.29%; it also reports Supernova beating Qwen3-0.6B on HellaSwag, PIQA, SuperGLUE, and SIQA, and beating Gemma 3 1B on MMLU, MMLU-PRO, and BBH. The explanation offered is threefold: the 128k English-tuned tokenizer compresses WikiText-103 to 4.78 characters per token, the 3:1 grouped-query attention cuts KV-cache memory by two-thirds, and the modern component set keeps every parameter productive. If this is right, the supposed need for trillions of training tokens and billion-plus parameters is partly an artifact of architectures and tokenizers not optimized for the small-model regime.

Load-bearing premise

The central 90% ratio rests on the Table 3 baseline numbers, and the paper gives no prompts, scoring details, sample counts, or seeds for the ten benchmarks, so a mismatched evaluation setup would make the headline comparison meaningless.

Editorial extensions

If this is right

  • A 650M model trained for 100B tokens can reach about 90% of a 1B model's zero-shot average on the ten benchmarks in Table 3.
  • Training cost falls by roughly two orders of magnitude: the paper estimates $10k and 960 GPU-hours versus about $1M and 370,000 hours for a 1B-class model.
  • Inference becomes cheaper and faster: the paper reports 59.6% higher throughput, 35.7% lower memory, and 37.5% lower latency than Llama 3.2 1B on A100 hardware.
  • A custom English-only tokenizer can outperform much larger multilingual vocabularies on English text, implying language-specific tokenization is a practical lever for English-centric products.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The tokenizer's contribution could be isolated by training the same 650M architecture with a generic multilingual tokenizer such as Llama's or GPT-4o's; if the benchmark average drops markedly, compression is the main driver, and if not, the architecture deserves the credit.
  • If the 90% ratio survives matched-harness evaluation, the paper's result suggests that compute-optimal scaling formulas fitted on larger models underestimate what careful small-model design can achieve, and 'tokens per parameter' should be treated as a tunable design variable for sub-billion models.
  • The English-only specialization points to a portfolio strategy: several per-language 650M models with language-specific tokenizers might deliver better cost-per-quality than one multilingual model for monolingual deployment, though the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes Supernova, a 650M-parameter decoder-only transformer that combines RoPE, GQA with a 3:1 grouping ratio, RMSNorm, SwiGLU, and a custom 128k-token byte-level BPE tokenizer. The central claims are that Supernova reaches about 90% of the average zero-shot performance of Llama 3.2 1B (43.09 vs. 47.73 in Table 3) with 35% fewer parameters and only 100B training tokens, and that its tokenizer achieves state-of-the-art compression (4.78 chars/token on WikiText-103). The paper also reports inference-efficiency metrics, a data-efficiency comparison, and cost estimates, and it concludes that architectural and tokenization efficiency can compensate for reduced parameter count and training data.

Significance. If the 90% / 100B-token claim were supported, it would be a notable counterexample to simple scaling-law expectations and would have practical value for low-resource training and deployment. The paper is clearly structured and provides a concrete architecture specification, and the tokenizer compression comparison on WikiText-103 is a potentially useful data point. The evidence, however, is currently far from the claim: the training-run description is internally contradictory, the evaluation harness for Table 3 is unspecified, and no code, checkpoints, or training logs are released. The headline numbers are point estimates without variance, and the paper offers no machine-checked proofs or reproducibility artifacts. The claimed result therefore cannot be considered established on the basis of this manuscript.

major comments (4)
  1. [5.2.1, 5.3.1] The training run is described with mutually incompatible numbers. Section 5.2.1 states a total of 1,000,000 training steps, while Section 5.3.1 states "14 days for 100,000 steps" and a throughput of about 300,000 tokens/s; the global batch is 983,040 tokens per step (480 sequences x 2048 tokens). At the stated batch size, 100B tokens corresponds to roughly 101,700 steps, which is consistent with the 100,000-step figure but not with the 1,000,000-step figure (about 983B tokens). Conversely, 100,000 steps at 300,000 tokens/s would take about 3.8 days, not 14, and 14 days at that throughput would consume about 363B tokens. No single correction to one of these numbers reconciles all three statements with the claimed 100B-token budget. Because the headline data-efficiency claim depends on the 100B-token total, and no training logs or checkpoints are provided, this inconsistency removes the evidential basis for the central comparison.
  2. [6.1.1, Table 3] The 90.29% headline ratio is computed from unweighted averages in Table 3, but the evaluation procedure is not described. The paper gives no prompt templates, no scoring functions or metric normalization details, no number of in-context examples, no sampling parameters, no seeds, and no per-benchmark sample counts. The baseline scores are presented as point estimates without variance or provenance, so it is impossible to know whether they were produced by the same harness and prompts as the Supernova scores. This makes the central performance comparison non-reproducible and the 90% claim unverifiable.
  3. [6.3, Table 6] The data-efficiency comparison is internally inconsistent. Table 6 lists Qwen3-0.6B as trained on 36,000B tokens, which at 600M parameters implies about 60,000 tokens per parameter, but the adjacent text says competitors range from about 1,636 to about 9,000 tokens per parameter. The same table lists Gemma 3 1B at 2,000B tokens and Llama 3.2 1B at 9,000B tokens, and the "Data Multiple" column presents these as exact multipliers, yet no sources are cited for any of these numbers. As a result, the claimed 18-360x data reduction and the tokens-per-parameter ratios are not supported by the table as printed.
  4. [5.4.1, Eq. (9)] Equation (9) is presented as an observed scaling law with fitted constants and exponents, but it is derived from a single training run at fixed N = 650M and D = 100B. Since only one (N, D) pair is reported, the equation reduces to a single constant and cannot validate a scaling relationship. No fitting procedure, additional data points, or model sizes are described. It does not directly determine the benchmark ratio, but the paper offers it as validation of the training dynamics, and it should either be supported with a proper fit over multiple scales or removed.
minor comments (6)
  1. [3.2] The component parameter counts in Section 3.2 sum to about 600.5M (196.6M embeddings + 177.4M attention + 226.5M FFN), not the stated approximately 650M; the discrepancy should be explained or corrected.
  2. [6.1.2, Table 3] The text says the average score is 43.10, while Table 3 reports 43.09; these numbers should be reconciled.
  3. [4.3.2] The special-token IDs jump from 4 to 8 with no explanation; the numbering should be made complete or the design choice should be documented.
  4. [5.4.1] The loss progression is summarized by initial and final values only, with no loss curve or convergence diagnostic shown; the claim of stable convergence and no overfitting is therefore not supported by a visual or tabular trace.
  5. [6.2.1, Table 4] The inference efficiency metrics report throughput, memory, latency, power, and cost per 1M tokens, but do not state batch size, sequence length, input/output length distribution, or the measurement protocol used on the A100 GPUs.
  6. [7.3.1, Table 7] The estimated training cost and CO2 figures for Llama 3.2 1B are presented without a source or calculation basis; a citation or explicit estimation formula is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline efficiency ratio is an external-benchmark comparison, and the only self-referential scaling fit is non-load-bearing.

full rationale

The central claims—Supernova's ~90% average zero-shot performance relative to Llama 3.2 1B, its tokenizer compression advantage, and its 100B-token training budget—are not derived from the model's own outputs or from the authors' prior work. Table 3 computes the 90.29% figure as a ratio of average scores over ten external benchmarks, and Table 1 compares the tokenizer against external tokenizers on the external WikiText-103 corpus; no fitted parameter is renamed as a prediction. Eq. (9) in Section 5.4.1 is an asserted scaling-law fit to the model's own loss curve, but it is not used to generate any benchmark result and therefore is not a load-bearing circular step; at most it is an unsupported empirical curve. There are no self-citations, imported uniqueness theorems, or ansatz-by-citation moves. The paper does have serious reproducibility and internal-consistency problems: Section 5.2.1 says 1,000,000 training steps while Section 5.3.1 says '14 days for 100,000 steps' at ~300,000 tokens/s, and the given 983,040-token batch makes 100B tokens equal ~101,725 steps, so these figures cannot all be true; Section 6.1 also omits prompt templates, seeds, and scoring details. These are verification/correctness failures, not circularity, and the benchmark comparison remains externally grounded in substance.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claim is an empirical measurement claim. It depends on unverified assumptions about data, training, and evaluation comparability, plus several hand-chosen architecture hyperparameters. No artifacts are provided to check any of these, and the training narrative is internally inconsistent.

free parameters (4)
  • Loss scaling law coefficients in Eq. (9) = 6.12, 138.7, 5.21, 0.39, 0.52
    Five constants fitted to the model's own training loss curve in Section 5.4.1. No derivation, no error bars, and the formula does not reproduce the claimed final loss of 2.14 at N=650M and D=100B.
  • Tokenizer vocabulary size = 128,000
    Chosen by hand with no ablation showing the effect of vocabulary size on benchmark performance.
  • GQA grouping ratio = 12 query heads / 4 KV heads (3:1)
    No ablation comparing 1:1, 2:1, 3:1, or 6:1; the claimed 1-2% quality impact is asserted without evidence.
  • Context length = 2048 tokens
    Fixed by design and not varied, which bounds both tokenizer comparisons and downstream evaluation.
assumptions (4)
  • domain assumption The benchmark scores in Table 3 are accurate and comparable across models.
    The central 90% claim assumes the listed Llama 3.2 1B, Qwen3, Gemma 3, and OpenELM scores come from comparable zero-shot evaluations. No prompts, harness, or scoring details are provided.
  • domain assumption The 100B-token Nemotron-CC subset was used with the described composition and quality filters.
    Section 5.1 asserts specific composition percentages and a five-step filtering pipeline, but no subset links, sampling code, or documentation are provided.
  • domain assumption The custom tokenizer achieves 4.78 characters per token on WikiText-103 under the same conditions as the comparison tokenizers.
    Table 1 reports speeds and compression ratios without specifying hardware or benchmark methodology for the comparison systems.
  • ad hoc to paper The training run described in Section 5 actually occurred with the reported loss trajectory and final loss.
    The paper gives contradictory step counts, durations, throughput, and a loss scaling equation that does not evaluate to the claimed final loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Supernova: Achieving More with Less in Transformer Architectures." pith.science (2026). https://pith.science/paper/5RWRKINF

@misc{pith2026250715773,
  author       = {Pith},
  title        = {Pith review of: Supernova: Achieving More with Less in Transformer Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5RWRKINF}},
  note         = {Machine review of arXiv:2507.15773}
}
read the original abstract

We present Supernova, a 650M-parameter decoder-only transformer that demonstrates how careful architectural design and tokenization innovation can achieve the performance of larger models while maintaining computational efficiency. Our architecture combines Rotary Positional Embeddings (RoPE), Grouped Query Attention (GQA) with a 3:1 compression ratio, RMSNorm for computational efficiency, and SwiGLU activation functions. A critical innovation is our custom 128,000-vocabulary byte-level BPE tokenizer, which achieves state-of-the-art compression performance. Through detailed analysis, we show that Supernova achieves 90% of the performance of 1B-parameter models while using 35% fewer parameters and requiring only 100B training tokens--an order of magnitude less than competing models. Our findings challenge the prevailing scaling paradigm, demonstrating that architectural efficiency and tokenization quality can compensate for reduced parameter counts.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 4 canonical work pages

  1. [1]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  3. [3]

    Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  4. [4]

    Roformer: Enhancedtransformer with rotary position embedding.arXiv preprint arXiv:2104.09864 , 2021

    JianlinSu, YuLu, ShengfengPan, BoWen, andYunfengLiu. Roformer: Enhancedtransformer with rotary position embedding.arXiv preprint arXiv:2104.09864 , 2021

    arXiv 2021

  5. [5]

    Gpt-neox-20b: An open-source autoregressive language model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745 , 2022

    arXiv 2022

  6. [6]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    arXiv 1911

  7. [7]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 , 2023

    arXiv 2023

  8. [8]

    Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    arXiv 2016

Show all 22 references
  1. [9]

    Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization.Advances in Neural Information Processing Systems, 32, 2019

    2019

  2. [10]

    Language modeling with gated convolutional networks.International Conference on Machine Learning , pages 933–941, 2017

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks.International Conference on Machine Learning , pages 933–941, 2017

    2017

  3. [11]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202 , 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202 , 2020

    2002 arXiv

  4. [12]

    Palm: Scaling language modeling with pathways.arXiv preprint arXiv:2204.02311 , 2022

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.arXiv preprint arXiv:2204.02311 , 2022. 24

    2022 arXiv

  5. [13]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725, 2016

    2016

  6. [14]

    Byte pair encoding is suboptimal for language model pre- training

    Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pre- training. Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 4617–4624, 2020

    2020

  7. [15]

    Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 66–71, 2018

    2018

  8. [16]

    Textbooks are all you need.arXiv preprint arXiv:2306.11644 , 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644 , 2023

    2023 arXiv

  9. [17]

    Stable lm 2 1.6b: Improving upon our previous language model with significantly improved training

    Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable lm 2 1.6b: Improving upon our previous language model with significantly improved training. arXiv preprint a...

    2024 arXiv

  10. [18]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295 , 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295 , 2024

    2024 arXiv

  11. [19]

    Scaling laws for neural language models

    JaredKaplan, SamMcCandlish, TomHenighan, TomBBrown, BenjaminChess, RewonChild, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  12. [20]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556 , 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556 , 2022

    2022 arXiv

  13. [21]

    Scaling data- constrained language models

    Niklas Muennighoff, Alexander Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Tan, Vishesh Patil, Tim Dettmers, Hyung Won Chung, Quentin Li, et al. Scaling data- constrained language models. arXiv preprint arXiv:2305.16264 , 2023

    2023 arXiv

  14. [22]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971 , 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971 , 2023. 25

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.