Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Study

Justice Sefas; Michael Walters; Rafael Kaufmann; Thomas Kopinski

REVIEW 4 major objections 5 minor 1 cited by

Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Study

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Gatekeeper vehicles that compute a free-energy risk metric make a mixed autonomous fleet safer, even when only a fraction of vehicles are gatekeeper-controlled.

desk verdict A clean, honest expected-utility gatekeeping demo whose 'positive externalities' claim isn't supported by the ego-only measurements. read the letter →

arxiv 2502.04249 v1 pith:EJ5LQZTY submitted 2025-02-06 cs.AI cs.LGcs.MAphysics.data-anstat.ML

Michael Walters , Rafael Kaufmann , Justice Sefas , Thomas Kopinski This is my paper · ORCID

classification cs.AIcs.LGcs.MAphysics.data-anstat.ML

keywords freeenergyprincipleactiveinferencecumulativeriskexposureautonomousvehiclesAIsafetymulti-agentsystemsgatekeeperpreferenceprior

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper repurposes the Free Energy Principle into a risk metric, the Cumulative Risk Exposure (CRE), and tests it as a 'gatekeeper' in a simulated autonomous vehicle fleet. A gatekeeper simulates possible futures, scores them against a stakeholder-defined preference prior, and switches a vehicle between an aggressive and a defensive driving policy when risk crosses a threshold. The central claim is that even when only a third of the ego vehicles are gatekeeper-controlled, collisions and overall loss improve across the whole fleet, and with full coverage the fleet approaches the defensive baseline's safety while keeping the aggressive baseline's speed. If true, safety could be improved by adding a modest number of risk-evaluating monitors rather than by controlling every agent or building a complete world model. The authors present this as a first VFE-based gatekeeper arrangement for agentic AI, with the metric intended as a transparent decision rule for risk governance.

What carries the argument

The central object is the Cumulative Risk Exposure (CRE), defined as $G_\Sigma(\phi,t)=\sum_{t'} \gamma^{t'} G_{t+t'}(\phi)$, where each step's risk $G_t$ is a free-energy expression combining an extrinsic 'energy' term (expected loss under the preference prior) and an entropic term weighted by inverse temperature $\beta$. The preference prior is built as a Boltzmann distribution over a stakeholder-defined loss function $L$, so $\beta$ acts as a preference temperature calibrated from the range of losses and stakeholder desirabilities. In the experiment the entropic term is dropped by assumption, leaving expected utility; gatekeepers run Monte Carlo rollouts, average the resulting risk over their local neighborhood, and use two hysteresis thresholds ($\rho^*_+=1.1\rho^*$, $\rho^*_-=0.9\rho^*$) to switch vehicles between the Hotshot and Defensive policies. This mechanism is what carries the argument: it converts a preference distribution into an online, decentralized intervention rule.

What would settle it

Replace the hand-built loss with a different plausible safety loss, or add observation noise to the gatekeeper's Monte Carlo rollouts; if the measured safety gains vanish or reverse, the result depends on the assumed loss and world model rather than on the free-energy metric itself.

Watch

Extended reading notes

Core claim

Starting from variational free energy, the paper derives a Cumulative Risk Exposure metric $G_\Sigma$ as a time-discounted expectation of the stakeholder's loss under a Boltzmann preference prior $\tilde p(L)=e^{-\beta L}/Z$. In a fully observable simulator the entropic terms drop out, so CRE reduces to time-discounted expected utility, and gatekeepers compute it with 128 Monte Carlo rollouts to a 10-step horizon. The experimental discovery is that gatekeeper-mediated policy switching—between a 'Hotshot' and a 'Defensive' driving policy, with decisions based on the average CRE of neighboring gatekeepers—produces measurably fewer collisions and lower loss than either fixed baseline, with the benefit increasing with the fraction of online vehicles. The paper interprets this as evidence that preference-encoded free energy can serve as an online risk signal that improves collective safety.

Load-bearing premise

The whole experiment assumes the hand-built loss function and the simulator's world model are exactly right, so the gatekeeper's risk numbers and the measured safety gains are only meaningful if that assumption holds.

Editorial extensions

If this is right

With only 4 of 12 ego vehicles under gatekeeper control, crashes and loss already improve measurably over the Hotshot baseline, so safety gains do not require every agent to be monitored.
At full gatekeeper coverage, the fleet tracks the defensive baseline's collision and defensive-driving performance while preserving most of the Hotshot speed reward, meaning the best-of-both-worlds outcome is empirically available.
Because the gatekeeper uses a single aggregate loss scalar rather than a full sensor suite, the same preference-encoding approach can be applied to other agentic systems where stakeholders can specify losses over outcomes.
Stakeholders can push collision rates lower by increasing the collision penalty in the loss function or updating the preference prior, giving a transparent tuning knob for risk governance.
In partially observable settings the full CRE with entropic and inverse-temperature terms becomes relevant; the current experiment only exercises the degenerate expected-utility form.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next test would compare CRE-triggered switching against random or fixed-schedule switching at the same rate; this would isolate whether the risk computation itself, rather than having a second policy available, drives the improvement.
If the positive-externality result transfers, retrofit deployments seem plausible: a minority of equipped vehicles could raise safety for unequipped traffic, which matters for mixed fleets during the transition to fully autonomous driving.
The Boltzmann preference prior gives a natural calibration procedure—setting $\beta$ from the stakeholder's desirability ratio between worst and best outcomes—that could carry over to real-world settings where losses have units such as cost or injury risk.
Neighborhood averaging of CRE suggests a route to collective safety: gatekeepers that share risk estimates can implement joint free-energy minimization, which in principle connects to cooperative inverse reinforcement learning and other joint-preference settings.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a well-written, honest paper whose headline claim is unsupported by its own measurement. The gatekeeper experiment shows that a policy-switching layer using time-discounted expected utility (they call it CRE) improves that same loss in a highway-env simulation. The math in Sections 1-2 is standard and correct, and they deserve credit for explicitly stating that, in a fully observable setup, CRE reduces to expected utility (Eq. 12) and that the entropic term is dropped ex hypothesi.

The main problem is the abstract's claim of 'significant positive externalities.' The dependent variables are ego-only: crashed counts come from '4 online ego vehicles' and 'Crashed' is defined as worlds with an ego crash. No crash statistics are reported for Alters or for the 8 offline Hotshot egos in the 4/12 condition. So the data can only support a private safety benefit to gatekeeper-controlled vehicles, not a system-level improvement. It's entirely possible that collisions are being shifted to untracked vehicles. The loss metric is likewise an ego aggregate. This isn't a minor gap — it's the central claim.

Related to that, because the gatekeeper optimizes the same loss used to measure success, the improvement is partly tautological. A stronger test would report collision rates for all vehicles, or use a separate safety metric not included in the loss function.

Other soft spots: no significance tests (the 'significant' in the abstract is descriptive), hand-picked threshold rho*=2 and reward constants with no sensitivity analysis, and only one simulator. The free-energy content is untested — the distinctively 'free energy' part (the entropic term) is dropped, so the paper is really about expected-utility gatekeeping.

That said, the paper is sincere and internally coherent. The authors flag their simplifications, the code is public, and the gatekeeper idea with neighborhood-averaged risk is a useful demonstration for the active-inference community. With revisions — measure all vehicles, add statistical tests, and scale back the externality claim — this could be a solid workshop or applied paper. As it stands, the abstract overreaches.

I'd send it to review because the topic is timely and the empirical setup is reproducible, but I'd expect the referee to demand major changes. My own verdict is skeptical on the headline, though the underlying experiment has legs.

Referee Report

4 major / 5 minor

Summary. The paper proposes a risk metric, Cumulative Risk Exposure (CRE), derived from variational free energy / active inference principles, and applies it to a multi-agent autonomous-vehicle simulation. Ego vehicles are optionally controlled by a gatekeeper that computes CRE via Monte Carlo rollouts and switches each online ego between 'Hotshot' and 'Defensive' policies when the neighborhood-averaged CRE crosses a threshold. The reward is a hand-built sum of speed, collision, and defensive-driving components; loss is its negative. The authors report that 4/12 and 12/12 gatekeeper penetration reduce loss and ego crashes relative to a Hotshot baseline, and they claim that low-penetration gatekeeping generates positive externalities for system safety. The paper also contains an extensive review of FEP/EFE/FEF theory and argues that the approach is more parsimonious than alternative safe-AI frameworks.

Significance. If the experimental claims were established, the result that a minority of gatekeeper-controlled vehicles can improve system-wide safety would be practically relevant. The manuscript has concrete strengths: the code is publicly available, the experimental protocol is described in enough detail to be reproduced, and the formal derivations in Sections 1 and 2 are standard and correctly presented. However, the evidence as reported does not support the externality claim, and the proposed risk metric collapses to expected utility under the assumptions made in Section 3.2, which substantially limits the theoretical novelty. The gatekeeping mechanism itself could still be a useful case study after reframing and additional analyses.

major comments (4)

[§3.2 / Eqs. (9)–(12)] Equation (12) makes explicit that, after dropping the entropic term, CRE is simply a time-discounted expected loss: GΣ = Σ γ^t E[L]. Because the gatekeeper's decision rule minimizes this same L and the paper's main evaluation metrics (Loss and Crashed) are derived from the reward components that define L, the reported safety improvement is largely a consequence of the construction. The paper itself concedes this in §3.2 ('CRE is identical to time-discounted expected utility'). To make the claim substantive, compare against a direct expected-utility policy-switch baseline that does not use the free-energy machinery, and report outcome metrics not embedded in L, such as all-vehicle collision counts or minimum time-to-collision.
[§3.3 / Fig. 1] The headline 'positive externalities' is not supported by the reported dependent variables. Section 3.3 states that only 4 online ego vehicles are tracked for collisions, and Fig. 1 defines Crashed as 'how many worlds have had an ego crash.' In the 4/12 condition this counts only 4 vehicle-policies per run, while the baselines count 12, so the cumulative crash curves are not exposure-normalized and may not be comparable across conditions. No statistics are reported for Alters or for the 8 uncontrolled Egos, so a shift of collisions from tracked online Egos to untracked vehicles would be invisible. The paper should report system-level statistics over all 24 vehicles, split by Ego/Alter and online/offline status, with exposure normalization.
[§3.4 / §3.5] The quantitative claim of 'significant' improvement is not backed by significance tests or robustness checks. The paper uses 1200 runs per configuration, but reports only averaged curves with 90% CIs; no bootstrap, permutation, or other test is given for the differences between Online-4, Online-12, and the baselines. The central hyperparameters (ρ*=2, α, σ, v_T, κ, λ, ζ, γ) are all heuristically chosen, and only one environment is used. Add statistical comparisons over the 1200 worlds and a sensitivity analysis for ρ*, κ, and λ; the threshold ρ* is especially important because it directly controls switching frequency.
[§2.2 vs. §3.2] Section 2.2 derives β from a stated range of loss values and stakeholder desirabilities, but Section 3.2 says the scale of β is irrelevant. This is only true for ranking policies with a fixed loss function; the intervention threshold ρ* is scale-dependent, and the paper chooses ρ*=2 without relating it to the loss scale. The relationship between β, the loss scale, and ρ* should be clarified, or the calibration discussion in §2.2 should be removed or substantially revised.

minor comments (5)

[Fig. 1] The Loss panel shows mean loss around 0.125–0.2, but §3.1 defines loss as the negative sum of rewards; if rewards are scaled in the displayed units, the sign or normalization is unexplained.
[Fig. 1] The Crashed panel would benefit from a precise definition of the y-axis (cumulative count of terminated worlds versus number of crashed vehicles) and from a statement of whether the 90% CI is across worlds or across Monte Carlo trajectories.
[Abstract and §3.3] The abstract and Section 3.3 contain the spacing artifact 'A V fleet' and 'A V experiment'; please use 'AV' consistently throughout.
[Eq. (12)] The notation in Eq. (12) uses E_{p(L)}[L] without defining the expectation variable; clarify whether the expectation is under the Monte Carlo rollout distribution p(o,x|π).
[§3.1 / References] Reference [SA23] is cited to support non-Markovian utility functions, but the paper's loss function is Markovian; either connect this more explicitly to the gatekeeper framework or remove the implication.

Circularity Check

2 steps flagged · score 6.0 of 10

CRE is definitionally expected loss, so the gatekeeper's measured safety gain is the same objective it was built to minimize.

self definitional [Section 3.2, Eq. (12) and Section 3.5]
"In this context, therefore, CRE is identical to time-discounted expected utility. ... GΣ(L) = − P τ t′ γ t′ Ep(L)[ln p̃(L)] = P τ t′ γ t′ Ep(L)[L] (12) ... The ultimate goal here is better decision-making according to stakeholder preferences through simulated futures. To that end, our main measuring stick is the defined loss L and collision results."

With the Boltzmann prior p̃(L) ∝ e^{−βL}, −ln p̃(L) = βL + ln Z, so Eq. (12) makes CRE a scalar multiple of expected loss L. Gatekeepers threshold this same expected loss to switch policies (Section 3.4), and the paper's evidence of 'increased system safety' is reduced L and collision counts, where collisions enter L through the reward RC. Thus the headline result reduces by construction to 'the gatekeeper lowers the loss it was explicitly designed to lower'; the free-energy machinery adds no independent content once entropy is dropped. The switching dynamics could still fail empirically, but the target and the metric coincide by definition.
renaming known result [Section 2 and Section 3.2]
"We propose and showcase an arrangement that repurposes and reframes the VFE construction laid out above. ... In this context, therefore, CRE is identical to time-discounted expected utility."

The paper presents Cumulative Risk Exposure as a novel FEP-derived risk metric and calls the gatekeeper arrangement 'the first VFE-based gatekeeper model,' yet Section 3.2 explicitly equates CRE with time-discounted expected utility after dropping all entropic terms. Since the Boltzmann preference prior is only an exponential reweighting of the hand-built loss L, the claimed first-principles derivation is a relabeling of negative expected loss; the novelty claim rests on naming rather than on a distinct quantity.

full rationale

The central circularity is that the evaluation metric and the optimized objective are the same quantity: Eq. (12) defines CRE as expected loss, the gatekeeper acts to reduce this expected loss, and Section 3.5 measures success with 'the defined loss L and collision results.' This is a partial circularity (score 6), not a fully forced one, because the threshold-switching mechanism could in principle have failed to reduce realized loss; however, the free-energy framing contributes no independently testable content beyond expected-utility minimization. A second, milder issue is the renaming of expected utility as a 'Cumulative Risk Exposure' metric, which is a labeling rather than a derivation. The self-citations ([KGT21], [KL23], [KL24]) and the Gaia Protocol affiliation are not load-bearing for the experiment, and no uniqueness theorem is imported. Separately, the 'positive externalities' claim is not supported by the reported data because only 4 online ego vehicles are tracked for collisions (Section 3.3), so benefits to Alters and uncontrolled Egos are unmeasured; this is a measurement-validity concern, not circularity, and does not affect the score.

Assumptions & free parameters 9 free parameters · 5 assumptions · 2 invented entities

The framework rests on standard variational inference machinery plus a set of untested domain assumptions. The experiment introduces many hand-chosen constants and a preference prior that is asserted sufficient without validation. No invented physical entities are proposed; the Gatekeeper is a conceptual component whose benefit is only demonstrated in simulation.

free parameters (9)

beta (preference temperature) = scale irrelevant in experiment
Introduced in Eq. (9) to shape the Boltzmann preference prior; calibratable from Lmax/Lmin and desirability ratio, but dropped when entropy is removed in Section 3.2.
rho* (risk threshold) = 2
Hand-selected threshold for switching to Defensive policy in Section 3.4; influences all gatekeeper results.
alpha (speed reward amplitude) = unspecified
Heuristic constant in Eq. (10).
sigma (speed reward width) = unspecified
Heuristic constant in Eq. (10).
v_T (target speed) = unspecified
Heuristic constant in Eq. (10).
kappa (collision penalty) = unspecified
Heuristic constant chosen to ascribe high disincentive in Section 3.1.
lambda (defensive reward scale) = unspecified
Heuristic constant in Eq. (11).
zeta (proximity penalty offset) = unspecified
Heuristic constant adding proximity penalty in Eq. (11).
gamma (discount factor) = unspecified
Discount in Eq. (8); value not stated in the text.

assumptions (5)

standard math Variational inference and KL divergence identities (Eqs. 1-6)
Background used to define VFE, EFE, and FEF; standard derivations, no errors found.
domain assumption The loss L and preference prior p_tilde are sufficient and accurate
Stated explicitly in Section 3.2: 'we assert ex hypothesi...'; without this the risk metric and safety improvement are not grounded in real preferences.
domain assumption The highway-env IDM/MOBIL simulator is a valid proxy for real traffic
The experiment uses a 'pared-back simulator' in Section 3; results are assumed to transfer to real AV fleets.
domain assumption Monte Carlo rollouts with N_MC=128 and horizon tau=10 provide unbiased estimates of expected loss
The MC approximation in Section 3.2 implicitly assumes sufficient sample size; no convergence checks are reported.
ad hoc to paper Stakeholder preferences are completely captured by the hand-chosen reward components in Section 3.1
The speed, collision, and defensive terms are constructed by the authors, not elicited from stakeholders or validated.

invented entities (2)

Gatekeeper (GK)
purpose: Evaluates agent policies against a hidden preference prior and triggers policy switches to control risky behavior
A new conceptual component; its benefit is only shown in the paper's own simulation, with no external falsifiable prediction.
Cumulative Risk Exposure (CRE)
purpose: Time-discounted free energy risk metric for scoring policies
The paper's proposed metric; in the experiment it reduces to expected utility, so its distinctive free energy content is not independently evidenced.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Study." pith.science (2026). https://pith.science/paper/EJ5LQZTY

@misc{pith2026250204249,
  author       = {Pith},
  title        = {Pith review of: Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJ5LQZTY}},
  note         = {Machine review of arXiv:2502.04249}
}

read the original abstract

We investigate the Free Energy Principle as a foundation for measuring risk in agentic and multi-agent systems. From these principles we introduce a Cumulative Risk Exposure metric that is flexible to differing contexts and needs. We contrast this to other popular theories for safe AI that hinge on massive amounts of data or describing arbitrarily complex world models. In our framework, stakeholders need only specify their preferences over system outcomes, providing straightforward and transparent decision rules for risk governance and mitigation. This framework naturally accounts for uncertainty in both world model and preference model, allowing for decision-making that is epistemically and axiologically humble, parsimonious, and future-proof. We demonstrate this novel approach in a simplified autonomous vehicle environment with multi-agent vehicles whose driving policies are mediated by gatekeepers that evaluate, in an online fashion, the risk to the collective safety in their neighborhood, and intervene through each vehicle's policy when appropriate. We show that the introduction of gatekeepers in an AV fleet, even at low penetration, can generate significant positive externalities in terms of increased system safety.

Figures

Figures reproduced from arXiv: 2502.04249 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Free Energy-Inspired Cognitive Risk Integration for AV Navigation in Pedestrian-Rich Environments
cs.RO 2025-07 conditional novelty 4.0 of 10

A free-energy-inspired cognitive risk signal adapts pedestrian forces and a GCN-SAC vehicle controller, improving safety and efficiency in a three-pedestrian shared-space simulation.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[7358]

Free-Energy Equilib- ria: Toward a Theory of Interactions Between Boundedly-Rational Agents

DOI: 10.1371/journal.pcbi.1008420. [Had+24] Dylan Hadfield-Menell et al. Cooperative Inverse Reinforcement Learning. Feb. 2024. DOI: 10.48550/arXiv.1606.03137. [Hyl+24] David Hyland et al. “Free-Energy Equilib- ria: Toward a Theory of Interactions Between Boundedly-Rational Agents”. In: ICML 2024 Workshop on Models of Human Feedback for AI Alignment. 2024...

arXiv doi:10.48550/arxiv.1606.03137 2024
[7667]

Gener- alised Free Energy and Active Inference

DOI: 10.1162/neco a 01354. [PF19] Thomas Parr and Karl J. Friston. “Gener- alised Free Energy and Active Inference”. In: Biological Cybernetics 113.5–6 (Dec. 2019), pp. 495–513. ISSN : 0340-1200, 1432-0770. DOI: 10.1007/s00422-019-00805-w. 8 [PPF22] Thomas Parr, Giovanni Pezzulo, and K. J. Fris- ton. Active Inference: The Free Energy Princi- ple in Mind, ...

arXiv doi:10.48550/arxiv.2309.01933 2019

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [7358]

Free-Energy Equilib- ria: Toward a Theory of Interactions Between Boundedly-Rational Agents

DOI: 10.1371/journal.pcbi.1008420. [Had+24] Dylan Hadfield-Menell et al. Cooperative Inverse Reinforcement Learning. Feb. 2024. DOI: 10.48550/arXiv.1606.03137. [Hyl+24] David Hyland et al. “Free-Energy Equilib- ria: Toward a Theory of Interactions Between Boundedly-Rational Agents”. In: ICML 2024 Workshop on Models of Human Feedback for AI Alignment. 2024...

arXiv doi:10.48550/arxiv.1606.03137 2024

[2] [7667]

Gener- alised Free Energy and Active Inference

DOI: 10.1162/neco a 01354. [PF19] Thomas Parr and Karl J. Friston. “Gener- alised Free Energy and Active Inference”. In: Biological Cybernetics 113.5–6 (Dec. 2019), pp. 495–513. ISSN : 0340-1200, 1432-0770. DOI: 10.1007/s00422-019-00805-w. 8 [PPF22] Thomas Parr, Giovanni Pezzulo, and K. J. Fris- ton. Active Inference: The Free Energy Princi- ple in Mind, ...

arXiv doi:10.48550/arxiv.2309.01933 2019