Pith. sign in

REVIEW 3 major objections 4 minor 4 references

P: A Universal Measure of Predictive Intelligence

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes that intelligence is measurable as the accuracy of an agent's predictions about its perceived world, formalized in a quantity called P1.1.

desk verdict A serious proposal with a load-bearing bug in Equation 2 that rewards anti-predictors, so it needs revision, not rejection out of hand. read the letter →

arxiv 2505.24426 v1 pith:AKN5R4BW submitted 2025-05-30 cs.AI

David Gamez This is my paper
classification cs.AI
keywords intelligencemeasurementpredictiveuniversaltestumweltKolmogorovcomplexityartificialgeneralratioscale
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that intelligence is best understood as predictive ability and turns that idea into a computable measure, P1.1, applicable to any agent in its ordinary perceived environment. Previous tests of intelligence either compare only humans, require separate test settings, or remain theoretical sums over all possible environments. P1.1 records the agent's predictions about its own internal umwelt states, checks them against later states, subtracts the score a random guesser would get, and weights the result by the complexity of the predictions and of the environments. Two demonstrations, a virtual maze agent and a neural time-series predictor, show the measure can be calculated on medium-sized systems and scales roughly linearly with the number of predictions. If the paper's central hypothesis is right, this gives the first practical universal measure that could rank humans, animals and AIs on a single ratio scale.

What carries the argument

P1.1, a measure of predictive intelligence built from internal state transitions. The load-bearing pieces are: (1) the umwelt-state transition framework, adapted from state-based analyses of consciousness; (2) Hellinger distance as a prediction-match score with random-guess subtraction; (3) Kolmogorov-complexity ratios as corrections for trivial predictions and redundancy across environments; and (4) a logarithm that makes simple and complex agents easier to compare. The measure is agnostic about how the agent predicts, whether by statistical tables, neural networks, or brains, so long as predictions and future states can be observed.

What would settle it

Compare P1.1 with independent intelligence rankings on a set of agents where prediction accuracy and judged intelligence are known to diverge; for example, a perfect finite-state lookup-table agent that scores near the theoretical maximum in a regular maze but performs at chance in novel variations would expose whether the measure rewards memorization rather than intelligence.

Watch

Extended reading notes

Core claim

The central claim is that universal intelligence can be defined operationally as aggregated prediction accuracy, formalized as P1.1. The agent experiences a perceived environment, or umwelt, with finitely many states; at each state transition it issues probability distributions over future states. Each prediction is scored with Hellinger distance, reduced by the expected score of a uniform random guess, summed over all states and times, multiplied by a Kolmogorov-complexity term that penalizes trivial predictions and double-counting of similar environments, and compressed by a logarithm. The paper claims this is the first practical universal measure that can be applied directly to an agent as it interacts with its normal environment, avoiding the test-versus-reference-environment gap that limits population-based measures such as IQ and g.

Load-bearing premise

The load-bearing premise is the paper's H1, that prediction is the most important component of intelligence; if prediction skill can be high in agents that clearly lack intelligence, P1.1 would measure prediction rather than intelligence.

Editorial extensions

If this is right

  • If H1 is correct, P1.1 allows direct comparison of humans, animals, and AIs in their reference environments without designing a common test that every population can take.
  • The same measure can index an agent's intelligence differently in different umwelts, turning the narrow-versus-general distinction into a quantifiable, environment-indexed spectrum.
  • P1.1 can serve as a feedback signal for systems that autonomously optimize their own predictive intelligence over time.
  • In concrete AI-safety settings, P1.1 can quantify whether an AI is more intelligent than humans in an umwelt relevant to causing harm, which is one condition for dangerous behavior.
  • The measure could be used to compare the strength of scientific theories by scoring the accuracy and quality of their predictions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • P1.1's dependence on finite umwelts means an anytime sampling version is a prerequisite for measuring real-world systems with effectively unbounded state spaces, including most natural environments.
  • If the measure gains traction, intelligence assessment shifts from designing tests to instrumenting internal states, making internal transparency a practical requirement for AI systems whose intelligence is to be measured.
  • Because Kolmogorov complexity must be approximated and the results depend on how predictions and umwelts are encoded, a standardized encoding convention would be needed before P1.1 values could be compared across laboratories.
  • A direct test of H1 would compare agents with matched P1.1 scores but very different goal-achievement or generalization abilities; large divergences would indicate that prediction alone is not a sufficient operationalization of intelligence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes P1.1, a universal measure of predictive intelligence. An agent's predictions about future states of its perceived environment, or umwelt, are compared with the states that actually occur; the resulting prediction matches are summed over all states and times, weighted by Kolmogorov-complexity ratios, and finally log-transformed. The formulation is grounded in hypotheses H1 and H2 (prediction is the most important component of intelligence; predictive intelligence varies with the number of accurate predictions), relativized to umwelts via H3.1, H4.1, and H5. Two feasibility experiments are reported: a maze-exploring agent with statistical predictions and an LSTM-based time-series predictor. Both agents achieve P1.1 values close to the theoretical maximum computed from the same definition, and the computation time scales roughly linearly. The paper claims that P1.1 is the first practical universal measure of intelligence that can be applied directly to an agent in its normal reference environment (Section 7.2).

Significance. If the central claim were established, P1.1 would be a substantial contribution: it would offer an environment-indexed, ratio-scale comparison of predictive intelligence across humans, animals, and AI systems, with a concrete algorithm and publicly available code. The paper is transparent about its hypotheses and provides reproducible experiments and scaling measurements, which are strengths. However, the empirical sections demonstrate only computability, not that P1.1 tracks intelligence, and the current definition of prediction match has a monotonicity defect that undermines even the narrower claim of measuring predictive accuracy. The contribution is therefore significant in conception but requires substantive revision before it can support the stated conclusions.

major comments (3)
  1. [5.3 (Eq. 2)] Equation 2 defines PM(P,U)=|(1-H(P,U))-(1-H(R,U))|=|H(R,U)-H(P,U)|. Because of the absolute value, PM is not monotone in prediction accuracy: it is zero when the prediction equals the random baseline R, but it rises again as the prediction becomes systematically worse than random. In a two-state umwelt, a perfect predictor with H(P,U)=0 receives PM=H(R,U)≈0.541, while an anti-predictor that always assigns probability 1 to the wrong state receives PM≈0.459; in a three-state umwelt the anti-predictor still receives about 0.350. This directly contradicts H2 (Section 3.7), which ties predictive intelligence to the number of accurate predictions. The section title 'Eliminating Random Guesses' is therefore misleading: random guessing is the minimum, not the zero point, and anti-correlated agents are rewarded. The maze and time-series agents only improve with learning, so the reported experiments never encounter this class of agents and the pathology is invisible in the results. Please either remove the absolute value (for example, define PM as (1-H(P,U))-(1-H(R,U)), allowing negative values for worse-than-random predictions) or give a principled reason why anti-predictors should receive positive intelligence.
  2. [3.7 and 7.2] The central claim that P1.1 measures intelligence rather than prediction accuracy rests entirely on H1, 'Prediction is the most important component of intelligence,' and H2, which identifies predictive intelligence with the number of accurate predictions. The paper supports H1 with illustrative arguments about predictive brains, planning, and intelligence tests, but it provides no independent evidence that prediction dominates other candidate components of intelligence, such as goal-directed reasoning, memory, or abstraction. The experiments in Section 6 do not test H1; they only show that P1.1 can be computed for two artificial agents. Because P is defined as a sum of prediction matches, the near-maximum values in Tables 4 and 5 are consistency checks with the definition, not validations of construct validity. To make the manuscript's claims proportional to its evidence, either validate H1 with an independent criterion (for example, showing that P rankings correlate with established intelligence measures in a reference population) or explicitly reframe the contribution as a measure of predictive accuracy rather than a universal measure of intelligence.
  3. [6.2 and 6.3] The time-series evaluation computes P1.1 on the entire data set after training on the training split and fine-tuning on the validation split; the paper states that 'P1.1 was calculated for the entire data set.' Consequently, the LSTM's near-maximum P1.1 values in Figure 6b may reflect memorization of the training data rather than predictive skill on novel data. The footnote acknowledging that overfitted networks can be 'highly intelligent in this data set' does not remove this issue. Since Section 6.3 presents these values as demonstrations that the algorithm could measure the intelligence of the agents, the evaluation protocol is too weak. A held-out test set should be used for the P1.1 calculation, or the conclusions should be restricted to computability and scaling, which are the claims the current experiments actually support.
minor comments (4)
  1. [5.5 (Eq. 3.1)] The ratio K(p)/L(p) is presented as a complexity weight, but no normalization is specified for arbitrary alphabets; with LZUTF8, K(p) is measured in bits while L(p) is a string length in characters, so the ratio can exceed 1. Please clarify the intended normalization and its effect on PMu.
  2. [5.7 (Eq. 5)] Setting P1.1=0 whenever PM≤1 collapses all agents with low prediction match onto the same value. Given the paper's claim of a ratio scale, this floor deserves explicit discussion, since it makes comparisons between weak agents meaningless on that scale.
  3. [5.2] The use of Student's t-test for continuous predictions yields a binary match (0 or 1) and is sensitive to the chosen p-value and sample size. A graded continuous measure would better reflect degrees of predictive accuracy; the paper acknowledges this only in passing.
  4. [6.2 (Table 3)] The hyperparameters (time window, number of models, LSTM units, epochs, batch size, p-value) are user-set, but no sensitivity analysis is reported. Since P1.1 values may depend on these choices, a brief robustness check would strengthen the feasibility claim.

Circularity Check

1 steps flagged · score 6.0 of 10

P's claimed status as a measure of intelligence is a restatement of the paper's own hypothesis H1, not a derived result.

  1. self definitional [Section 3.7 (H1, H2) followed by Section 5.3-5.7, especially Eq. 2 and Eq. 5]
    "H1. Prediction is the most important component of intelligence. If H1 is correct, the intelligence of agents will vary with their ability to make accurate predictions: H2. Predictive intelligence varies with the number of accurate predictions that an agent can make."

    P1.1 is defined (Eqs. 3.1, 4, 5) as a logarithmic function of accumulated prediction match PM, i.e., of prediction accuracy. The only bridge from PM to intelligence is H2, which stipulates by definition that predictive intelligence varies with the number of accurate predictions. Therefore the paper's central claim that P1.1 is a universal measure of intelligence does not derive or test H1; it simply restates it under a new name. The experiments show that P1.1 can be computed and that agents can approach the theoretical maximum of this same formula, but that is a consistency check, not independent evidence that prediction accuracy constitutes intelligence.

full rationale

The central circularity is that the paper's measure is an operationalization of its own hypothesis H1/H2. H1 asserts that prediction is the most important component of intelligence; H2 then defines predictive intelligence as varying with the number of accurate predictions. The algorithm's P1.1 is built directly from prediction-match sums, so any statement that P1.1 measures intelligence is true by construction only if H1 is granted. The paper presents H1 with illustrative arguments, not with an independent measure of intelligence against which P1.1 is validated. The maze and time-series experiments demonstrate feasibility, scaling, and proximity to internally computed maxima, but they do not provide external validation of the intelligence interpretation. The self-citations to Gamez (2023) and Gamez and Aleksander (2011) are not load-bearing for the central claim. A separate, non-circularity concern is that Eq. 2's absolute value makes PM non-monotone in prediction accuracy, so systematically wrong predictors can receive nearly as much credit as perfect predictors; this is an internal correctness risk of the measure rather than a circularity of the derivation chain.

Assumptions & free parameters 7 free parameters · 8 assumptions · 1 invented entities

The central measure rests on five hypotheses that are assumed rather than validated, on a compression-based approximation of Kolmogorov complexity, on statistical choices for scoring predictions, and on a finite-state restriction. The experiments use these assumptions but do not test them against independent intelligence criteria.

free parameters (7)
  • p-value for prediction match = 0.05
    Threshold for Student's t-test in continuous predictions; determines whether a prediction counts as a match, directly affecting P scores in time-series experiments.
  • Time window = 3
    Number of past points fed to LSTM; affects agent's predictive accuracy in the demonstration, not the P formula.
  • Number of models/networks = 5
    Ensemble size for time-series predictions; produces a normal distribution of predictions.
  • LSTM units per model = 20
    Capacity of each network; influences prediction quality.
  • Number of epochs = 10
    Training length for the neural networks.
  • Batch size = 10
    Training batch size.
  • Log cutoff for P = PMU <= 1 maps to 0
    Equation 5 sets predictive intelligence to zero when total prediction match is at or below 1; this arbitrary threshold defines the zero point of the scale.
assumptions (8)
  • domain assumption H1: Prediction is the most important component of intelligence.
    Stated in Section 3.7; the entire measure operationalizes this hypothesis, with no independent evidence.
  • domain assumption H2: Predictive intelligence varies with the number of accurate predictions.
    Stated in Section 3.7; links the measure to prediction count.
  • domain assumption H3.1: An agent's intelligence varies with its umwelt.
    Section 4.3; intelligence scores are indexed to sets of umwelts.
  • domain assumption H4.1: Valid intelligence tests require measurements in reference umwelts.
    Section 4.3; used to argue against separate test environments.
  • domain assumption H5: Intelligence operates in a perceived environment or umwelt.
    Section 4.3; justifies the state-transition approach using umwelt states.
  • domain assumption Kolmogorov complexity can be approximated by LZUTF8 compression.
    Sections 5.5 and 5.6; the incomputable K is replaced by a specific compressor, which the paper acknowledges must be carefully chosen.
  • ad hoc to paper Prediction accuracy for discrete distributions is measured by Hellinger distance, and by Student's t-test for continuous variables.
    Section 5.2; these choices are not derived from prior results, and the t-test binarizes continuous matches into 0 or 1.
  • domain assumption Umwelts have finite numbers of states.
    Section 5.4; the algorithm enumerates all states and state transitions, excluding continuous and infinite umwelts.
invented entities (1)
  • Internal prediction states P^{u}_{s-t}
    purpose: Allow P to measure predictions by comparing internal state encodings with later umwelt states, avoiding the need for external tests.
    Section 5.1 introduces these states; for large opaque AIs and biological brains the paper admits current technology cannot reliably identify them, so there is no falsifiable handle outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of P: A Universal Measure of Predictive Intelligence." pith.science (2026). https://pith.science/paper/AKN5R4BW

@misc{pith2026250524426,
  author       = {Pith},
  title        = {Pith review of: P: A Universal Measure of Predictive Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKN5R4BW}},
  note         = {Machine review of arXiv:2505.24426}
}
read the original abstract

Over the last thirty years, considerable progress has been made with the development of systems that can drive cars, play games, predict protein folding and generate natural language. These systems are described as intelligent and there has been a great deal of talk about the rapid increase in artificial intelligence and its potential dangers. However, our theoretical understanding of intelligence and ability to measure it lag far behind our capacity for building systems that mimic intelligent human behaviour. There is no commonly agreed definition of the intelligence that AI systems are said to possess. No-one has developed a practical measure that would enable us to compare the intelligence of humans, animals and AIs on a single ratio scale. This paper sets out a new universal measure of intelligence that is based on the hypothesis that prediction is the most important component of intelligence. As an agent interacts with its normal environment, the accuracy of its predictions is summed up and the complexity of its predictions and perceived environment is accounted for using Kolmogorov complexity. Two experiments were carried out to evaluate the practical feasibility of the algorithm. These demonstrated that it could measure the intelligence of an agent embodied in a virtual maze and an agent that makes predictions about time-series data. This universal measure could be the starting point for a new comparative science of intelligence that ranks humans, animals and AIs on a single ratio scale.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

4 extracted references · 2 canonical work pages

  1. [1]

    D., Hasenclever, L., Klibaite, U., Gellis, A., Tassa, Y., Wayne, G., Botvinick, M

    Aldarondo, D., Merel, J., Marshall, J. D., Hasenclever, L., Klibaite, U., Gellis, A., Tassa, Y., Wayne, G., Botvinick, M. and Olveczky, B. P. (2024). A virtual rodent predicts the structure of neural activity across behaviors. Nature 632: 594-602. Balduzzi, D. and Tononi, G. (2008). Integrated information in discrete dynamical systems: motivation and theo...

    arXiv 2024

  2. [18]

    and Hutter, M

    Legg, S. and Hutter, M. (2007a). A Collection of Definitions of Intelligence. Proceedings of Advances in Artificial General Intelligence Concepts, Architectures and Algorithms: Proceedings of the AGI Workshop 2006, edited by B. Goertzel and P. Wang, IOS Press, pp. 17-24. Legg, S. and Hutter, M. (2007b). Universal intelligence: A definition of machine inte...

    work page 2007

  3. [88]

    Naglieri, J. A. and Bornstein, B. T. (2003). Intelligence and achivement: just how correlated are they? Journal of Psychoeducational Assessment 21: 244-60. Neisser, U., Boodoo, G., T. J. Bouchard, J., Boykin, A. W., Brody, N., Ceci, S. J., Halpern, D. F., Loehlin, J. C., Perloff, R., Sternberg, R. J. and Urbina., S. (1996). Intelligence: Knowns and Unknow...

    arXiv 2003

  4. [137]

    Tjøstheim, T. A. and Stephens, A. (2022). Intelligence as Accurate Prediction. Review of Philosophy and Psychology 13: 475-99. Uexküll, J. v. (2010). A Foray into the Worlds of Animals and Humans, with A Theory of Meaning. Translated by J. D. O'Neil. Minneapolis: University of Minnesota Press. Warwick, K. (2000). QI: The Quest for Intelligence. London: Pi...

    work page 2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.