{"id":"5e8f20c5-6eb6-4362-95fb-63c261e019c5","arxiv_id":"2505.04075","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The authors introduce a compute-dependent versus compute-independent framework and report nanoGPT experiments showing compute-independent algorithms such as LayerNorm and RoPE give compute-equivalent gains up to 1.9x, while compute-dependent ones become neutral only as model size grows.","lead":"This paper sorts LLM algorithmic improvements into two groups: ones that help at every computing budget and ones that only pay off with massive compute. It argues that hardware controls can slow AI progress but cannot stop algorithmic progress entirely, because the first group keeps improving small models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 contradicts the paper's own CEG rule: full-model MQA and sparse attention have lower minimum loss than baseline yet are reported with CEG<1, so the key compute-dependent validation is internally inconsistent.","rationale":"The paper's central claim is not only that some algorithmic advances are compute-independent; it is that the compute-dependent/independent distinction is empirically validated by the nanoGPT experiments. Table 3 is the only direct quantitative evidence, and the full-model rows for MQA and sparse attention are the linchpin of the compute-dependent side of that validation: they are what convert 'detrimental at small scale' into 'improving toward baseline with scale.' If those rows are internally inconsistent with the paper's own metric definition, the empirical validation fails before any extrapolation to frontier models is considered. The reader's scale-generalization concern is real and is honestly conceded in the Limitations section, but it is a second-order extrapolation issue; the Table 3 arithmetic problem is first-order and checkable from the stated formulas and the public repository. I would keep the reader's CONDITIONAL verdict rather than move to ACCEPT, because the qualitative framework, the case-study CEG estimates, and the compute-independent experimental results (LayerNorm, RoPE, FlashAttention, combined) may still survive; the required revision is to reconcile the measurement definition with the reported numbers and rerun or re-report the affected rows. I do not see grounds to REJECT, since the authors are transparent about scale and convergence limits and the framework has independent grounding in the cited scaling-law literature.","tokens_in":18376,"tokens_out":7890,"duration_ms":74492,"concrete_test":"Pull the saved validation-loss curves from the linked GitHub repository (or rerun the full-model MQA and sparse-attention configs with the same seeds) and apply the paper's stated primary rule literally: set L_target to the baseline's validation loss at iteration 50,000; find the first iteration S_alg where the algorithm's validation loss is <= L_target; compute CEG = 50,000 / S_alg. Then verify whether the Table 3 values 0.931 (MQA) and 0.964 (sparse attention) are reproduced under any documented variant, such as using final loss rather than minimum, or extrapolating S_alg beyond 50,000. Also record whether the 'Min. Val. Loss' column reports the training minimum or the final value. If the recomputed CEGs are >= 1.0, Table 3 and the 'improving to neutral' claim require correction; if the reported values reproduce, the paper must clarify the measurement rule that produces them.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is internal consistency of the experimental metric. The Experiments section defines primary CEG = S_base / S_alg, where S_alg is the first step at which an algorithm reaches L_target, the baseline's loss at its 50,000th iteration, and states that CEG > 1 means the algorithm reached L_target in fewer steps than the baseline. Under this rule, if an algorithm's validation loss ever drops to or below L_target within 50,000 steps, its CEG is necessarily at least 1.0. Table 3 reports the full-model baseline 'None' row with minimum validation loss 3.542 and CEG 1.000. The same table reports full-model MQA minimum validation loss 3.535 and sparse attention minimum validation loss 3.418, both below the baseline value, yet reports CEGs of 0.931 and 0.964. Those values are impossible under the primary rule, since S_alg <= 50,000 forces CEG >= 1. They also cannot come from the auxiliary rule, which applies only when an algorithm fails to reach L_target; a minimum below L_target is direct evidence that it did reach it. Therefore either the min-loss column, the CEG column, or the stated measurement definition is wrong. These two rows are precisely the evidence for the paper's headline claim that compute-dependent advances move from detrimental at small scale to 'on par with the baseline' at larger scale, so the quantitative validation of the central framework is not trustworthy until the raw loss curves are checked.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework that classifies algorithmic advancements in LLM development as compute-dependent (benefits emerge primarily at high compute) or compute-independent (benefits appear across scales), and uses Compute-Equivalent Gain (CEG) to quantify these effects. It presents case studies for several well-known algorithms (Transformer, Sparse Attention, MoE, MQA as dependent; RoPE, FlashAttention, LayerNorm as independent) and reports small-scale nanoGPT experiments at two model sizes (~50M and ~110M parameters) trained for 50,000 iterations. Based on these experiments, it claims that compute-independent advances yield performance gains across scales (e.g., combined CEG up to 3.5x), while compute-dependent advances are detrimental at small scale but improve toward baseline as scale increases. The paper concludes that hardware controls can slow but not halt algorithmic progress, and recommends that AI governance incorporate algorithmic research oversight.","tokens_in":18637,"tokens_out":5374,"duration_ms":52935,"significance":"The compute-dependent/independent distinction is a clear and potentially policy-relevant conceptual contribution. The case studies synthesize CEG estimates from the literature, and the authors make their code available, which supports reproducibility. If the experimental results were robust, the framework would offer a practical tool for forecasting AI progress and assessing governance interventions. However, the quantitative validation is currently fragile: it relies on a small scale range, single runs without error bars, and contains an apparent internal inconsistency in the reported CEG values. The policy conclusions are plausible and align with prior literature (e.g., Gupta et al. 2024), but they are not yet strongly supported by the empirical evidence presented here.","major_comments":[{"comment":"There is an internal inconsistency between the stated CEG definition and the values in Table 3 for the full-model rows. The paper defines primary CEG as S_base / S_alg, where S_alg is the first step at which the algorithm reaches L_target, the baseline's validation loss at its 50,000th iteration. Therefore, if an algorithm's validation loss ever falls to or below L_target within those 50,000 steps, its CEG must be at least 1.0. Table 3 reports full-model MQA and Sparse Attention with minimum validation losses of 3.535 and 3.418, both below the baseline's minimum of 3.542, yet the table lists CEGs of 0.931 and 0.964. Since these minima are direct evidence that the algorithms reached L_target, the auxiliary rule (which is reserved for algorithms that fail to reach L_target) does not apply. Either the min-loss column, the CEG column, or the stated measurement definition is wrong. These two rows are the key evidence for the claim that compute-dependent advances move from detrimental at small scale to 'on par with the baseline' at larger scale, so the quantitative validation is not trustworthy as reported.","section":"Experiments, Table 3 and CEG definition"},{"comment":"All CEG values come from a single training run per configuration, with no seeds or error bars. Because CEG is derived from the first step at which the loss crosses L_target, a single noisy low-loss spike can produce a spuriously high CEG, and the absence of repeated runs makes it impossible to assess whether the reported compact-model values (e.g., 0.515 for Sparse Attention) or the combined 3.5x gain are reliably different from noise. The authors should rerun each configuration with multiple seeds (or at least report variance/confidence intervals) and make the raw loss curves available so that the step-to-target computations can be verified.","section":"Experiments, all reported results"},{"comment":"The experimental validation is at risk of circularity because the classification of an algorithm as compute-dependent is defined by where its benefits materialize. Observing low small-scale CEG for MQA and Sparse Attention is thus partly a restatement of the classification rather than an independent confirmation. The case studies in Table 2 provide some independent grounding, but the experimental test would be substantially stronger if the classifications were pre-registered or if the experiments covered a wider, pre-specified range of scales to test the predicted scaling trend. As it stands, the results are consistent with the framework but do not strongly discriminate it from alternative explanations.","section":"Methodology, 'Our Framework' and Results"},{"comment":"The central policy conclusion—that hardware restrictions are insufficient to prevent all capability gains—assumes that the compute-dependence behavior observed at 50M–110M parameters persists at frontier scales. The paper's own Limitations section concedes that the modest scale and lack of full convergence 'could influence the precise CEG values reported and the observed performance characteristics of some algorithms.' Without larger-scale evidence or a theoretical argument explaining why the observed trend should extrapolate, the empirical basis for the policy claim is thin. The authors should either soften the policy statements or provide additional evidence, such as experiments at a few more scales or a connection to established scaling-law frameworks.","section":"Implications for Policymakers and Limitations"}],"minor_comments":[{"comment":"The phrase 'fit will within our framework' appears to be a typo; it should read 'fit well within our framework.'","section":"Methodology, after Eq. (2)"},{"comment":"The x-axis of Figure 1 is labeled up to 40 (thousands of iterations), but the text in the Experiments section states that training ran for 50,000 iterations. Please clarify whether the curves are truncated or the axis is mislabeled.","section":"Figure 1"},{"comment":"For MQA, the CEG column lists encoder and decoder speedups rather than a single CEG value; the table would be clearer if it noted explicitly that these are inference-time speedups and distinguished them from the FLOP-based CEG defined in Eq. (2).","section":"Table 2"},{"comment":"The reference 'Andrej 2025' is incomplete; the name should be given in full (Andrej Karpathy) and the GitHub repository URL should be included, as done in the abstract.","section":"References"},{"comment":"The hypothesis that 'the attention mechanism is the primary compute consumer' is speculative given the small set of algorithms tested; it should be framed as a conjecture to be tested in future work.","section":"Discussion"},{"comment":"The phrase 'combined CEG up to 3.5x' is derived from a single run without error bars; the wording should be qualified to reflect the limited evidence, e.g., 'in our single-run experiments, the combined algorithms achieved CEG ≈ 3.5x.'","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper tackles a timely and policy-relevant question and offers a useful conceptual framework. However, the experimental validation is central to the paper's quantitative claims, and the apparent contradiction in Table 3 must be resolved before the paper can be considered reliable. I recommend requesting the raw loss curves and a corrected analysis. If the corrected analysis still supports the qualitative conclusions, the paper could become acceptable for a venue that welcomes policy-oriented AI research; as it stands, the load-bearing evidence is not yet sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one for the framework, not for the numbers. The core distinction—compute-dependent advances pay off at scale, compute-independent ones pay off everywhere—is a useful organizing lens for AI governance and forecasting, and the case studies do a good job of connecting that lens to concrete algorithms with rough CEG estimates from the original papers. That synthesis alone is worth the read.\n\nThe nanoGPT experiments are the weak part. The stress-test note is right: under the paper's own primary CEG rule, if an algorithm ever reaches the baseline's final loss within 50,000 steps, its CEG must be at least 1.0. But Table 3 reports full-model MQA and sparse attention with minimum losses below the baseline (3.535 and 3.418 vs 3.542) yet CEGs of 0.931 and 0.964. That is impossible under the stated definition, and it cannot be rescued by the auxiliary rule because those minima directly prove the target was reached. These two rows are exactly the evidence for the headline claim that compute-dependent advances improve toward parity as scale increases. So the quantitative validation collapses until that table is reconciled with the actual loss curves.\n\nEven setting that aside, the experimental support is thin: no seeds, no error bars, single runs, only 50M and 110M parameters, and training cut off at 50k iterations. The authors do flag the scale and convergence limits in their Limitations section, which earns them credit, but they do not flag the Table 3 inconsistency, which is more serious. The circularity concern is real but secondary—since compute-dependence is partly defined by where benefits materialize, seeing low small-scale CEG for those algorithms is consistency, not independent confirmation. Still, the framework has enough independent grounding in the literature that this is fixable in revision.\n\nWho gets value: governance people, forecasters, and anyone who wants a vocabulary for asking whether an algorithmic improvement transfers across compute scales. The policy conclusion—hardware controls slow but do not stop algorithmic progress—is plausible and consistent with the qualitative evidence, but it currently overreaches relative to the experimental validation.\n\nRecommendation: send it to peer review, but with a request for the raw loss curves, seeds, and error bars, and a requirement that Table 3 be reconciled or the experimental section rewritten to avoid the contradiction. If that is fixed, this becomes a citable reference for the compute-dependence concept; until then, cite it for the framework and not for the measured CEG values.","headline":"A genuinely useful compute-dependent/independent framework with a solid literature synthesis, but the empirical validation as reported has an internal contradiction in Table 3 that needs fixing before the experimental claims can be trusted.","tokens_in":19172,"tokens_out":1623,"would_cite":true,"duration_ms":18458,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that LLM gains split into compute-dependent and compute-independent algorithmic advances, and that compute-independent advances keep producing measurable progress even when hardware is restricted.","keywords":["compute-equivalent gain","compute-dependent advancements","compute-independent advancements","algorithmic progress","LLM governance","hardware controls","scaling laws","nanoGPT"],"falsifier":"Run the same fixed-loss CEG protocol on a compute-dependent algorithm such as sparse attention across a wider range of model sizes, from 100M to 1B parameters or beyond; if its CEG stays below 1 at the larger sizes, or if a compute-independent algorithm such as RoPE loses its advantage above some scale, the classification and the governance conclusion would fail.","tokens_in":18154,"feed_emoji":"⚙️","tokens_out":8429,"duration_ms":77549,"temperature":0.7,"pith_summary":"The paper's central claim is that algorithmic progress in large language models comes in two distinct modes: compute-dependent advances, whose benefits materialize mainly when training compute is large, and compute-independent advances, which improve efficiency at every compute scale. The authors quantify each advance with compute-equivalent gain (CEG), the factor by which an improved algorithm reduces the compute needed to reach a baseline performance level. In nanoGPT experiments at 50M and 110M parameters, compute-independent methods such as layer normalization, rotary position embeddings, and FlashAttention improved efficiency at both scales, with the three combined reaching CEG up to $3.5\\times$; compute-dependent methods (multi-query attention, sparse attention) hurt at the smaller scale and approached parity at the larger scale. The policy conclusion is that hardware restrictions can slow frontier development but cannot prevent algorithmic capability gains, so governance should also target algorithmic research. This matters because export controls and compute thresholds are currently the main regulatory levers for frontier AI.","feed_headline":"Compute controls can't fully halt LLM progress, tests show","feed_subtitle":"Efficiency algorithms give up to 3.5x compute-equivalent gains without extra hardware.","key_machinery":"The central object is compute-equivalent gain (CEG), defined as $C_b/C_e$, the ratio of compute (FLOPs) a baseline model needs to reach a target loss to the compute an improved model needs to reach the same loss. The classification scheme around it is the load-bearing mechanism: an algorithm is compute-independent if it beats the status quo by a similar amount at baseline and increased compute, and compute-dependent if its advantage is absent or negative at baseline but grows with compute. The experiments operationalize CEG by fixing FLOPs per step for each model size and counting steps to a fixed validation-loss target, so a CEG above 1 means fewer FLOPs for the same loss.","core_discovery":"The discovery is a way to separate two types of algorithmic progress and evidence that the separation holds in small models. Advances are classified by comparing their performance gain over a predecessor at baseline and at increased compute: if the gain is similar at both, it is compute-independent; if it is small or negative at baseline and large at higher compute, it is compute-dependent. Using CEG as the common yardstick, the paper estimates that compute-dependent advances are historically the largest (transformer $20$--$50\\times$, mixture-of-experts $7$--$11\\times$, sparse attention $4.8$--$7\\times$), while compute-independent advances are smaller but universal (RoPE $1.7\\times$, layer normalization $1.67\\times$, FlashAttention formally $1\\times$ with a $2$--$4\\times$ practical speedup). The experiments reproduce this pattern: at 50M parameters, LayerNorm and RoPE give CEGs of $1.836\\times$ and $1.870\\times$ and a combined $3.483\\times$, while multi-query attention and sparse attention give $0.673\\times$ and $0.515\\times$; at 110M parameters, the compute-dependent methods rise to $0.931\\times$ and $0.964\\times$, consistent with benefits that emerge as compute grows.","pith_inferences":["A testable extension of the paper's framework is to apply CEG to data-quality and inference-time improvements; the authors list these as out of scope, but the same ratio would reveal whether their compute-independence classification holds beyond pretraining.","The paper's observation that its compute-dependent examples all alter the attention mechanism implies a prediction it does not prove: the next very large CEGs will come from attention-cost changes such as linear or recurrent attention, while non-attention components yield smaller gains.","If the scale-transfer assumption is correct, monitoring published small-scale efficiency gains could serve as a leading indicator for frontier progress, because compute-independent advances should propagate upward at little extra cost.","The combined-gain result raises the possibility that the paper's single-algorithm CEG estimates understate real progress; frontier models bundle many advances, so their joint CEG could be larger than the sum of the parts."],"forward_implications":["Hardware export controls should be expected to slow frontier LLM progress rather than stop it, because compute-independent algorithms keep improving models within any fixed compute budget.","The largest per-advance gains remain tied to high compute, so restricting large-scale training would still suppress the most significant capability jumps.","If compute-independent results transfer across scale, small models become a cheaper testbed for algorithmic search, which should make automated discovery of such advances more effective.","Compute thresholds used as governance tripwires become less reliable, since algorithmic gains can push a model's capability across a threshold without crossing the corresponding compute threshold.","Algorithmic gains can combine super-additively, as the combined LayerNorm+RoPE+FlashAttention CEG of $3.5\\times$ exceeds the individual effects, so capability assessments should test bundles of advances rather than single changes."],"supporting_citations":[{"why":"Supplies the compute-equivalent gain (CEG) measure that the paper adapts as its central quantitative tool.","marker":"Davidson et al. 2023"},{"why":"Defines the transformer, the paper's primary compute-dependent case, and provides the training-cost numbers used for a $20$--$50\\times$ CEG estimate.","marker":"Vaswani et al. 2017"},{"why":"Introduces rotary positional embeddings (RoPE), the paper's main compute-independent case, with the BERT/RoFormer comparison used for a $1.7\\times$ CEG estimate.","marker":"Su et al. 2023"},{"why":"Introduces FlashAttention and reports the $2$--$4\\times$ training speedups that ground its compute-independent classification.","marker":"Dao et al. 2022"},{"why":"Introduces layer normalization and reports the 60% training-step result used for a $1.67\\times$ CEG estimate.","marker":"Ba, Kiros, and Hinton 2016"},{"why":"Introduces sparse attention, a compute-dependent case, and supplies the EnWiki8 results used for a $4.8\\times$ CEG estimate.","marker":"Child et al. 2019"},{"why":"Introduces multi-query attention, the compute-dependent case whose decoding speedup and small-scale perplexity trade-off drive classification.","marker":"Shazeer 2019"},{"why":"Provides the Switch Transformer $7\\times$ training speedup used as the mixture-of-experts CEG estimate.","marker":"Fedus, Zoph, and Shazeer 2022"},{"why":"Provides the DeepSeek-V3 case study showing near-frontier performance at a fraction of reference training compute, motivating the hardware-controls argument.","marker":"DeepSeek-AI et al. 2024"},{"why":"Quantifies algorithmic progress in language models and supplies the estimate that the transformer accounts for roughly 20% of gains, used to sanity-check the transformer CEG.","marker":"Ho et al. 2024"}],"fun_headline_variants":["LLM gains via algorithms even with compute limits: 3.5x CEG","Compute curbs can't stop LLM progress from algorithmic wins","Algorithm tweaks beat hardware limits in LLM efficiency tests","Study: efficiency boosts offer LLM compute-free gains up to 3.5x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the compute-dependence pattern observed at 50M and 110M parameters also holds at the much larger scales of real frontier models, which the paper does not test.","fun_headline_variants_meta":{"raw":{"variants":["LLM gains via algorithms even with compute limits: 3.5x CEG","Compute curbs can't stop LLM progress from algorithmic wins","Algorithm tweaks beat hardware limits in LLM efficiency tests","Study: efficiency boosts offer LLM compute-free gains up to 3.5x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1784,"prompt_tokens":1034,"completion_tokens":750,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":670}},"tokens_in":650,"tokens_out":750,"duration_ms":7842,"temperature":1.0,"reasoning_tokens":670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:38:51.168052+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fixed-loss CEG protocol on a compute-dependent algorithm such as sparse attention across a wider range of model sizes, from 100M to 1B parameters or beyond; if its CEG stays below 1 at the larger sizes, or if a compute-independent algorithm such as RoPE loses its advantage above some scale, the classification and the governance conclusion would fail.","supporting_citations":[{"cited_title":"N.; Kaiser, .; and Polosukhin, I","cited_arxiv_id":null,"evidence_quote":"Defines the transformer, the paper's primary compute-dependent case, and provides the training-cost numbers used for a $20$--$50\\times$ CEG estimate."}],"review_version":1}