Preparing for the Intelligence Explosion

Fin Moorhouse; William MacAskill

REVIEW 3 major objections 4 minor 4 references

Preparing for the Intelligence Explosion

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that AI substituting for human researchers could compress a century of technological progress into less than a decade, so the resulting 'grand challenges' need preparation now, not deferral to superintelligence.

desk verdict A genuinely useful reframing of AI preparedness, but the 'century in a decade' number is not as solid as the main text suggests — the authors themselves include a model (footnote 58) under which it fails, and never resolve it. read the letter →

arxiv 2506.14863 v1 pith:G46JJRGS submitted 2025-06-17 cs.CY cs.AI

William MacAskill , Fin Moorhouse This is my paper · ORCID

classification cs.CYcs.AI

keywords intelligenceexplosionAIresearcheffortgrandchallengesideaproductionfunctiontechnologicalprogressalignmentAGIpreparednesssemi-endogenousgrowth

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that AI systems that can substitute for human researchers will probably drive a century's worth of technological progress in less than a decade, because AI research effort is already growing more than 600 times faster than human research effort and scaling trends show no imminent halt. That compressed century would present a rapid sequence of consequential, hard-to-reverse decisions, which the authors call 'grand challenges': new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, digital minds with moral standing, and opportunities to improve collective decision-making. The paper's central policy claim is that these challenges cannot always be delegated to a future aligned superintelligence, because some arise before superintelligence exists, some have preparation windows that close early, and some require human institutions to be improved in advance. The paper therefore makes the case for a broader 'AGI preparedness' agenda, beyond alignment alone, with concrete steps available today.

What carries the argument

The central object is the semi-endogenous idea production function $\dot{A}_t / A_t = \alpha S_t^\lambda A_t^{-\beta}$, where $A_t$ is the technology level and $S_t$ the number of effective researchers; the paper treats AI research effort as directly adding to $S_t$. Using historical estimates of $\lambda = 0.75$ and $\beta = 2.4$, the paper derives that sustaining a research-effort growth rate of about one doubling per year yields a century of progress in a decade, and that a 600-fold increase in total research effort is the required threshold. This function is the bridge that converts trends in training compute, algorithmic efficiency, and inference compute into a quantitative prediction about the pace of technological progress.

What would settle it

Track total factor productivity growth and the ratio of effective AI-to-human research effort. If AI-equivalent research effort increases by the projected factor of 600 or more over a decade while TFP growth stays near its historical 1.25% per year, the substitution claim would be empirically falsified. A more targeted check: measure whether AI research effort shows rapidly diminishing returns in parallel (with $\lambda$ falling well below 0.75) by testing whether the growth of AI-generated research output tracks the number of AI researcher instances or grows far more slowly.

Watch

Extended reading notes

Core claim

The central claim is that, on a default path of continued AI scaling, collective AI cognitive labour will reach parity with human research labour within roughly two decades, and then grow by factors of $10^7$ to $10^{14}$ over the following decade. Plugging that growth into an idea production function in which research output depends on total research effort $S_t$ as $\dot{A}_t / A_t = \alpha S_t^\lambda A_t^{-\beta}$ with $\lambda = 0.75$ and $\beta = 2.4$, the paper computes that total factor productivity would rise by an amount equivalent to more than 300 years of historical progress in ten years. The authors conclude that a century of technological progress in a decade is more likely than not, that this would likely be followed by an 'industrial explosion' of self-replicating robotic production, and that the resulting grand challenges warrant preparation now rather than deferral to aligned superintelligence.

Load-bearing premise

The argument depends on AI cognitive labour plugging into the same idea-production function as human researchers, with the same 'ideas get harder to find' and 'more researchers step on each other's toes' elasticities, so that an AI researcher is as productive at generating new ideas as a human researcher.

Editorial extensions

If this is right

On the default scenario in which AI scaling continues without a collective agreement to slow down, a century's worth of technological progress in a decade is likely, beginning soon after AI research effort reaches human parity.
A technology explosion would feed into an 'industrial explosion' of self-replicating robotic factories once robots substitute for human manual labour, removing the human bottleneck on industrial growth.
Grand challenges will arrive in rapid succession, including AI takeover, highly destructive technologies, power-concentrating mechanisms, value lock-in, digital minds, space governance, and epistemic disruption.
Many challenges cannot be punted to aligned superintelligence because they arise before it, have preparatory windows that close early, or involve time lags such as training human decision-makers.
Useful preparation now includes preventing extreme concentration of power, empowering responsible decision-makers, building AI tools for collective decision-making, and starting institutional design for digital minds and space governance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the paper's substitution claim: if the ratio of AI-equivalent to human research effort reaches the 600-fold threshold within a decade without TFP growth accelerating to roughly ten times its historical rate, the production-function parameters would be falsified.
If the 'stepping on toes' elasticity $\lambda$ for AI researchers is closer to 1 than to 0.75, as the paper notes would follow if that effect partly reflects declining average human researcher ability, then the research-effort threshold required for a century in a decade drops substantially, making the conclusion more robust.
The paper's framework implies that the share of research output that depends on serial physical experiments or human trials is the key measurable bottleneck; tracking that share over time would calibrate how much the 10-fold headwind adjustment the authors add actually matters.
An implicit tension in the paper is that 'slowing the intelligence explosion' and 'bringing superintelligence earlier' are both endorsed as preparedness strategies under different conditions; making that trade-off explicit suggests a portfolio of pacing measures and front-loading measures rather than a single stance.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this paper if you care about AI governance. It makes the case that AGI preparedness is not just alignment: a fast takeoff would bring a broad set of 'grand challenges' — new weapons, autocratic lock-in, digital minds, space governance, epistemic disruption — and many can't be punted to future aligned superintelligence. The 'punt vs prepare' timing analysis in Section 5 is the real contribution, especially the veil-of-ignorance argument for why some agreements are only possible now. The writing is clear and the paper is honest about many uncertainties.

The quantitative core is an idea production function borrowed from Bloom et al., with AI research effort substituted for human researchers. On that model, 5x/year growth in AI effort after parity gives 'a century in a decade.' But the authors themselves flag, in footnote 58, an alternative specification (g_A = θ ln S) under which the case 'no longer seems clear' — and indeed it pushes cumulative progress below the century benchmark. The main text's response is a qualitative 'further 10x' fudge factor, not a resolution of that alternative. That is a real soft spot. The paper would be stronger if it either presented sensitivity bounds across specifications or downgraded the headline claim to 'very rapid progress, plausibly decades per decade.'

The qualitative argument, though, survives the quantitative wobble. Even if progress is only 30 years in a decade, the grand challenges remain and the point about early windows of opportunity still holds. The citation pattern is largely fine; the MacAskill co-authored citations are for background premises, not the load-bearing result. There are no invented entities. The main missing piece is a proper sensitivity analysis and a clearer demarcation of which premises come from prior work by the authors.

This paper deserves peer review — not desk rejection. A serious referee would ask for sensitivity analysis and a more honest treatment of footnote 58, but the synthetic contribution is real. I'd bring it to our reading group, and I'd probably cite the 'punt vs prepare' framing. My verdict: conditional accept, with revision.

Referee Report

3 major / 4 minor

Summary. The paper argues that if AI systems become able to substitute for human research labor, the resulting growth in total research effort could compress a century of technological progress into about a decade. It models this via a semi-endogenous idea production function, estimates that AI research effort could grow 5x per year or faster after human-AI parity, and concludes that a 'century in a decade' is likely on the default scaling path. The paper then catalogs 'grand challenges'—including AI takeover, destructive technologies, power concentration, value lock-in, digital minds, space governance, and epistemic disruption—and argues that many cannot be deferred to aligned superintelligence, proposing near-term preparation measures.

Significance. If the central quantitative claim were robust, the paper would make an important contribution to AI governance discussions by broadening the agenda beyond alignment to include a wide range of fast-moving societal challenges. The paper is unusually transparent for a policy-adjacent essay: it states its model, names parameters (lambda = 0.75, beta = 2.4, gamma = 0.7), gives explicit growth-rate scenarios, and even flags a competing model in footnote 58. It also offers concrete, falsifiable claims about compute and efficiency trends and a useful taxonomy of governance challenges. However, the quantitative centerpiece is not yet robust: the paper's own alternative log-research specification undermines the headline conclusion, and the main text's extra '10x' allowance for physical bottlenecks is unmodeled.

major comments (3)

[Section 3, 'The technology explosion', footnote 58 and following paragraph] The log-research model introduced in footnote 58 is not resolved, and it directly undercuts the headline claim. With g_A = theta ln(S) calibrated to a current elasticity of 0.75 and g_A = 1.5%, multiplying S by 10^7 gives an end-of-decade growth rate of about 0.196, but the cumulative TFP multiplier over the decade is only about e^{1.06} ≈ 2.9, below the paper's own 'century in a decade' benchmark of e^{1.25} ≈ 3.5. Adding the fishing-out term A^{-beta} with beta = 2.4 would reduce this further. The authors concede that under this specification 'the case no longer seems clear,' yet the main text proceeds to assert that 'a century's worth of technological progress in a decade seems likely.' The manuscript needs either a defense of the power-function specification over the log specification, a presentation of both as scenarios with distinct conclusions, or a substantial weakening of the headline claim.
[Section 3, text after footnote 66] The 'further 10x increase in cognitive research effort' used to absorb physical-experiment and capital bottlenecks is not derived. Footnote 66's Cobb-Douglas adjustment with gamma = 0.7 implies that cognitive effort must grow by a factor of about 3.73 more than in the gamma = 1 case, not by a factor of 10. The additional factor of roughly 2.7x is an unmodeled fudge. This matters because the conservative scenario's margin above the century-in-a-decade threshold is enormous under the power-function model but shrinks dramatically under the log model of footnote 58; the unmodeled 10x is therefore load-bearing for the conclusion and should be either derived or removed.
[Section 3, equation dA/dt / A = alpha S^lambda A^{-beta} and footnote 57] The central calculation transplants Bloom et al.'s elasticities lambda = 0.75 and beta = 2.4, estimated for human researchers, to AI cognitive labor without justification. If AI researchers have different 'stepping on toes' dynamics—for example, because they can be copied, parallelized, or coordinated more easily—or if AI research faces different 'fishing out' dynamics, then the factor-of-600 threshold and the 'three hundred years in a decade' conclusion change substantially. The paper's own footnote 58 is one illustration of this sensitivity. A sensitivity analysis over lambda and beta, together with an argument for why the human-researcher elasticities apply to AI labor, is needed to support the quantitative claim.

minor comments (4)

[Footnote 58] The calibration of the log model is under-explained: the reader must reverse-engineer why theta = 0.01125 and ln(S_0) = 4/3 follow from requiring a current elasticity of 0.75 and a current growth rate of 1.5%. A one-line derivation would make the competing model much easier to evaluate.
[Section 2 and footnote 57] The phrase 'century in a decade' is used in two different senses: Section 2's historical thought experiment compresses all scientific, technological, political, and philosophical developments of 1925-2025, while the formal definition in footnote 57 is a century of 1.25% annual TFP growth. This ambiguity makes it easy for a reader to overstate what the model establishes.
[Footnote 66] The equation in footnote 66 contains an incomplete phrase: 'Assuming no growth in [P]' appears to be missing the variable P (physical labor/capital). This should be fixed.
[Footnote 68] The dismissal of Almeida, Naudé, and Sequeira as making 'an error in its calculations' is asserted without specifics. If this is meant to preempt a contrary source, the error should be identified precisely.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the central 'century in a decade' claim is a parameterized extrapolation from external empirical estimates, with only minor non-load-bearing self-citations.

full rationale

The paper's central derivation is not equivalent to its inputs. The idea production function dA/dt / A = α S^λ A^{-β} uses λ = 0.75 and β = 2.4 from Bloom et al., and the 600x research-effort threshold for a century in a decade is obtained by algebra from that external model plus the definition of a century as the equilibrium 1.25%-per-year TFP path — not by fitting the target result. The AI research effort growth estimates (25x per year, conservative 5x per year after parity) come from independently cited compute, efficiency, and inference trends (Epoch AI, METR, Anthropic), and the conclusion is robust in the paper's own conservative scenario even if the software feedback loop never materializes. Self-citations to Davidson, Hadshar, and MacAskill appear in support of the rapid feedback-loop scenario and upper bounds on algorithmic efficiency, but those are not load-bearing for the main conservative-scenario conclusion, so they do not make the central claim reductionist. The paper itself flags a genuine robustness limitation, not a circularity: footnote 58 concedes that replacing the power function with g_A = θ ln(S) calibrated to the same current elasticity leaves the case 'no longer seems clear,' especially with fishing-out restored. That is a model-uncertainty caveat about an assumed functional form, not a case of a prediction being defined in terms of the input or a fitted parameter being renamed as a prediction. No uniqueness theorem from the authors' prior work is invoked to forbid alternative specifications, and no known empirical pattern is merely renamed with new coordinates. Accordingly, the paper is self-contained against external benchmarks for the purpose of circularity assessment, with only minor self-citation present.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

No new physical or formal entities are postulated. 'Grand challenges', 'software feedback loop', and 'digital minds' are conceptual categories or future contingencies, not invented existents.

free parameters (7)

lambda (research elasticity with respect to researcher count) = 0.75
Borrowed from Bloom et al.; used in the idea production function dA/dt / A = alpha S^lambda A^(-beta) (Section 3, 'The technology explosion'). The century-in-a-decade requirement of roughly 600x research-effort growth is directly sensitive to lambda.
beta (fishing-out elasticity) = 2.4
Bloom et al.'s revised estimate for lambda=0.75; controls how much research productivity declines with cumulative technology. Used in footnote 57 to translate 5x per year AI research effort growth into 'more than 300 years' of progress.
gamma (cognitive labor share in research) = 0.7
From NSF R&D labor share; used in the Cobb-Douglas extension in footnote 66 to require 43% faster cognitive effort growth when physical factors are static.
Training compute growth rate = 4.5x/year
Epoch AI estimate (footnote 12); drives the 25x/year AI research effort estimate. The paper notes the 90% CI spans 1.5x-11.8x.
Algorithmic training efficiency growth rate = 3x/year
Ho et al. estimate; combined with compute to give about 10x/year effective training compute growth.
Inference compute growth rate = 2.5x/year
Frymire and Owen estimate of deployed accelerator FLOP/s; combines with efficiency to give about 25x/year 'AI population' growth.
Post-training enhancement efficiency growth rate = 3x/year
Anthropic informal estimate (footnote 16); raises effective compute growth to about 30x/year and AI research effort to about 75x/year in the aggressive case.

assumptions (5)

domain assumption Semi-endogenous growth model (idea production function) describes how research effort translates into technological progress.
Invoked in Section 3 'The technology explosion'; the entire century-in-a-decade calculation uses dA/dt / A = alpha S^lambda A^(-beta) with parameters from Bloom et al.
domain assumption AI cognitive labor substitutes for human researcher labor in the production function at parity, with equivalent marginal products.
The paper models AI research effort as adding to S_t in researcher-equivalents (footnote 25 and surrounding text); physical experiments, capital, and serial bottlenecks are treated as secondary in Section 3.
domain assumption Historical exponential trends in compute, algorithmic efficiency, and inference scale continue at roughly current rates for the coming decade.
Section 3 'Progress in AI capabilities' and scenario tables; the 25x/year and 600x-faster claims depend on these trend extrapolations, which the paper acknowledges carry wide uncertainty.
standard math If a doubling of AI research inputs yields at least a doubling of outputs, a self-sustaining software feedback loop is possible.
Section 3 'How far could AI keep improving?'; the paper relies on empirical efficiency studies to assign 'roughly even chance' to this condition, but treats it as a premise.
domain assumption The value framework: avoiding existential catastrophe and improving the expected value of Earth-originating life is the relevant decisional criterion.
Section 4 defines 'grand challenges' via a 0.1% expected-value threshold; this is a longtermist ethical premise, adopted without argument.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Preparing for the Intelligence Explosion." pith.science (2026). https://pith.science/paper/G46JJRGS

@misc{pith2026250614863,
  author       = {Pith},
  title        = {Pith review of: Preparing for the Intelligence Explosion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G46JJRGS}},
  note         = {Machine review of arXiv:2506.14863}
}

read the original abstract

AI that can accelerate research could drive a century of technological progress over just a few years. During such a period, new technological or political developments will raise consequential and hard-to-reverse decisions, in rapid succession. We call these developments grand challenges. These challenges include new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, and digital beings worthy of moral consideration, as well as opportunities to dramatically improve quality of life and collective decision-making. We argue that these challenges cannot always be delegated to future AI systems, and suggest things we can do today to meaningfully improve our prospects. AGI preparedness is therefore not just about ensuring that advanced AI systems are aligned: we should be preparing, now, for the disorienting range of developments an intelligence explosion would bring.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 4 linked inside Pith

[4]

Rethink Priorities

https:/ / epoch.ai/blog/how-predictable-is-language-model-benchmark-performance . Rethink Priorities. ‘How Bad W ould Nuclear Winter Caused by a US-Russia Nuclear Exchange Be?’, 20 June 2019 . https:/ /rethinkpriorities.org/research-area/how-bad-would-nuclear-winter- caused-by-a-us-russia-nuclear-exchange-be/ . Rose, E. M. The Murder of William of Norwich...

arXiv doi:10.48550/arxiv.2012.08347 2019
[2022]

Kahan, Dan M., Ellen Peters, Erica Cantrell Dawson, and Paul Slovic

https:/ /papers.ssrn.com/sol3/papers.cfm?abstract_id=4201147 . Kahan, Dan M., Ellen Peters, Erica Cantrell Dawson, and Paul Slovic. ‘Motivated Numeracy and Enlightened Self-Government’. Behavioural Public Policy 1, no. 1 (May 2017): 54–86. https:/ /doi. org/10.1017/bpp.2016.2 . Karger, Ezra, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fre...

arXiv doi:10.48550/arxiv.2501.16946 2017
[2023]

Chan, Alan, Kevin W ei, Sihao Huang, Nitarshan Rajkumar, Elija Perrier, Seth Lazar, Gillian K

https:/ / coordination.substack.com/p/ alignment-is-not-enough . Chan, Alan, Kevin W ei, Sihao Huang, Nitarshan Rajkumar, Elija Perrier, Seth Lazar, Gillian K. Hadfield, and Markus Anderljung. ‘Infrastructure for AI Agents’. arXiv, 17 January 2025. https:/ / doi.org/10.48550/ arXiv.2501.10114 . Committee on the Peaceful Uses of Outer Space. ‘UNISPACE IV -...

arXiv doi:10.48550/arxiv.2501.10114 2016
[2024]

Bales, Adam

https:/ /situational-awareness.ai/superalignment/#The_problem . Bales, Adam. ‘ Against Willing Servitude: Autonomy in the Ethics of Advanced Artificial Intelligence - Adam Bales’. Global Priorities Institute (blog), 28 October 2024. https:/ / globalprioriti esinstitute.org/ against-willing-servitude-autonomy-in-the-ethics-of-advanced-artificial- intellige...

arXiv doi:10.48550/arxiv.2311.08379 2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [4]

Rethink Priorities

https:/ / epoch.ai/blog/how-predictable-is-language-model-benchmark-performance . Rethink Priorities. ‘How Bad W ould Nuclear Winter Caused by a US-Russia Nuclear Exchange Be?’, 20 June 2019 . https:/ /rethinkpriorities.org/research-area/how-bad-would-nuclear-winter- caused-by-a-us-russia-nuclear-exchange-be/ . Rose, E. M. The Murder of William of Norwich...

arXiv doi:10.48550/arxiv.2012.08347 2019

[2] [2022]

Kahan, Dan M., Ellen Peters, Erica Cantrell Dawson, and Paul Slovic

https:/ /papers.ssrn.com/sol3/papers.cfm?abstract_id=4201147 . Kahan, Dan M., Ellen Peters, Erica Cantrell Dawson, and Paul Slovic. ‘Motivated Numeracy and Enlightened Self-Government’. Behavioural Public Policy 1, no. 1 (May 2017): 54–86. https:/ /doi. org/10.1017/bpp.2016.2 . Karger, Ezra, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fre...

arXiv doi:10.48550/arxiv.2501.16946 2017

[3] [2023]

Chan, Alan, Kevin W ei, Sihao Huang, Nitarshan Rajkumar, Elija Perrier, Seth Lazar, Gillian K

https:/ / coordination.substack.com/p/ alignment-is-not-enough . Chan, Alan, Kevin W ei, Sihao Huang, Nitarshan Rajkumar, Elija Perrier, Seth Lazar, Gillian K. Hadfield, and Markus Anderljung. ‘Infrastructure for AI Agents’. arXiv, 17 January 2025. https:/ / doi.org/10.48550/ arXiv.2501.10114 . Committee on the Peaceful Uses of Outer Space. ‘UNISPACE IV -...

arXiv doi:10.48550/arxiv.2501.10114 2016

[4] [2024]

Bales, Adam

https:/ /situational-awareness.ai/superalignment/#The_problem . Bales, Adam. ‘ Against Willing Servitude: Autonomy in the Ethics of Advanced Artificial Intelligence - Adam Bales’. Global Priorities Institute (blog), 28 October 2024. https:/ / globalprioriti esinstitute.org/ against-willing-servitude-autonomy-in-the-ethics-of-advanced-artificial- intellige...

arXiv doi:10.48550/arxiv.2311.08379 2024