pith. sign in

arxiv: 2606.21015 · v1 · pith:MTVLO4DYnew · submitted 2026-06-19 · 💻 cs.AI

The AI Evaluability Gap: The Missing Layer for Managing Risk and Sustaining Value

Vishal Srivastava , Tanmay Sah This is my paper

Pith reviewed 2026-06-26 14:35 UTC · model grok-4.3

classification 💻 cs.AI
keywords AI governanceevaluability gapevidence sufficiencyoperational certificationinvestment certificationAI risk managementAI value sustainmentevidence properties
0
0 comments X

The pith

The AI Evaluability Gap arises because governance focuses on system properties instead of the evidence needed to justify decisions about those properties.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that organizations deploying AI encounter an evaluability gap where they lack enough evidence for confident choices on managing risks and maintaining value. This occurs due to an overemphasis on attributes like safety and fairness without ensuring the supporting evidence has the right qualities. The authors propose Evaluability as the system's ability to produce and update such evidence, defined by six properties including observability and verifiability. They separate operational certification for deployment from investment certification for funding. Readers should care because unresolved, this gap prevents reliable AI oversight in organizations.

Core claim

The paper establishes that the AI Evaluability Gap represents a category error in AI governance, where attention to system properties such as safety, fairness, reliability, compliance, and value overshadows the need for adequate evidentiary foundations. It defines Evaluability as the capability of a system to generate, maintain, and renew evidence sufficient to support high-confidence governance decisions over time. Governance decisions are formalized as functions of calibrated confidence Conf(D|E), and evidence must satisfy six properties: observability, attributability, intervenability, verifiability, calibration, and temporal validity. The framework separates Operational Certification rel

What carries the argument

Evaluability, defined as the capability of a system to generate, maintain, and renew evidence sufficient to support high-confidence governance decisions over time.

If this is right

  • Governance decisions depend on evidence meeting the six properties to achieve calibrated confidence.
  • Operational decisions about system deployment rely on structural evidence while investment decisions about resource allocation rely on causal evidence.
  • Addressing evidence sufficiency closes the gap and enables both risk management and value sustainment in AI organizations.
  • The six properties—observability, attributability, intervenability, verifiability, calibration, and temporal validity—form the basis for sufficient evidence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Adopting this approach might require organizations to implement new monitoring systems focused on tracking these evidence properties throughout an AI system's lifecycle.
  • The distinction between operational and investment certification suggests that different types of evidence audits could be needed depending on whether the decision is about launching or continuing funding.
  • This framework could extend to non-AI systems facing similar evidence challenges in governance contexts.

Load-bearing premise

The six properties of evidence are both necessary and jointly sufficient to close the AI Evaluability Gap and support high-confidence governance decisions.

What would settle it

An empirical study showing that organizations using evidence satisfying all six properties still fail to achieve high-confidence decisions, or that high-confidence decisions are possible without one or more of the properties.

Figures

Figures reproduced from arXiv: 2606.21015 by Tanmay Sah, Vishal Srivastava.

Figure 1
Figure 1. Figure 1: The Evaluability Gap. Governance of an AI system encompasses two classes of decisions: operational decisions drawing on risk-, fairness-, reliability-, and compliance-oriented evidence, and investment decisions drawing on value-, adoption-, productivity-, and business-impact evidence. Both feed a single governance function. The Evaluability Gap is the typical condition in which the available evidence is in… view at source ↗
Figure 2
Figure 2. Figure 2: The missing layer. Governance decisions depend on confidence; confidence depends on evidence; evidence [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Operational vs. Investment Certification. Both certifications rest on the same evaluability substrate, but they [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The Evaluability Cycle: Questions Every Governance System Must Answer. Each property of evaluable evidence answers a distinct governance question, and together they form a continuous cycle rather than an indepen￾dent checklist. Observability answers what happened? by capturing outcomes, decisions, behaviors, and impacts. Attributability answers did the AI cause it? through causal reasoning and experimental… view at source ↗
Figure 5
Figure 5. Figure 5: Evaluability as a Continuous Capability. The same six properties, viewed as a closed cycle rather than a sequence. The radial connections to the center indicate that Evaluability is not any single property but the joint capability of an organization to sustain all six over time. Where [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The evidence lifecycle. Continuous re-certification is not a procedural overlay but a structural requirement: in [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Evidence Decay and the Expiration of Governance Warrant. The vertical axis is the time-indexed Evaluability-Based Evidence Score, EBES(t) = Conf(D | E(t)), defined in Section 3.1. The evidence supporting a governance decision decays at different rates across AI systems as users adapt, models retrain, markets evolve, and capabilities emerge; the horizontal threshold τ is the minimum EBES required for the de… view at source ↗
Figure 8
Figure 8. Figure 8: Investment Certification Requires Both Value and Evidence. The vertical axis is the Evaluability-Based Evidence Score, EBES = Conf(D | E), defined in Section 3.1; for the use cases shown, D is an investment decision (continue, scale, divest, investigate). Two use cases with identical estimated business value may warrant different governance actions depending on the EBES supporting those estimates: it captu… view at source ↗
read the original abstract

Organizations deploying AI face two fundamental governance challenges: managing AI risk and sustaining AI value. Both depend on evidence whose sufficiency cannot be taken for granted. We call the shared underlying challenge the AI Evaluability Gap: the condition in which organizations lack sufficient evidence to support high-confidence governance decisions regarding either risk or value. We argue that this gap reflects a category error in current practice. Existing governance approaches focus primarily on properties of systems, such as safety, fairness, reliability, compliance, and value, while paying comparatively little attention to the evidentiary foundations required to justify decisions about those properties. We further argue that AI governance encompasses both operational decisions regarding whether a system may operate and investment decisions regarding whether it merits continued organizational resources. To address this problem, we introduce Evaluability, defined as the capability of a system to generate, maintain, and renew evidence sufficient to support high-confidence governance decisions over time. We formalize governance decisions as functions of calibrated confidence Conf(D|E) and identify six properties of evaluable evidence: observability, attributability, intervenability, verifiability, calibration, and temporal validity. The framework distinguishes Operational Certification, which relies primarily on structural evidence to justify deployment decisions, from Investment Certification, which relies primarily on causal evidence to justify continued resource allocation. We argue that evidence sufficiency is a missing layer of AI governance and that closing the AI Evaluability Gap is a prerequisite for both managing risk and sustaining value in AI-enabled organizations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that organizations deploying AI face an 'AI Evaluability Gap'—a lack of sufficient evidence to support high-confidence governance decisions on risk management and value sustainment. It argues this reflects a category error, with existing approaches over-focusing on system properties (safety, fairness, etc.) rather than evidentiary foundations. The authors introduce 'Evaluability' as the capability to generate, maintain, and renew evidence, formalized via calibrated confidence Conf(D|E), and enumerate six properties of evaluable evidence: observability, attributability, intervenability, verifiability, calibration, and temporal validity. The framework distinguishes Operational Certification (structural evidence for deployment) from Investment Certification (causal evidence for resource allocation), positioning evidence sufficiency as a missing governance layer.

Significance. If the framework holds, it supplies a conceptual distinction between structural and causal evidence that could clarify governance trade-offs in AI deployment versus continued investment. The paper's explicit separation of operational and investment certification decisions is a clear organizational contribution. However, as a definitional and argumentative piece without empirical measurements, case studies, or formal derivation of the six properties, its significance rests on whether the framework is adopted and tested by practitioners rather than on demonstrated predictive or prescriptive power.

major comments (2)
  1. [Abstract and framework section] Abstract and framework section: The six properties of evaluable evidence are introduced by enumeration without derivation from the Conf(D|E) function or demonstration that they are individually necessary and jointly sufficient to support high-confidence governance decisions. This is load-bearing for the central claim that closing the evaluability gap addresses a category error in current practice.
  2. [Abstract] Abstract: The assertion that current governance approaches 'pay comparatively little attention to the evidentiary foundations' is presented without supporting measurements, citations to specific frameworks, or case studies showing systematic neglect; this underpins the category-error diagnosis but remains untested within the manuscript.
minor comments (1)
  1. The distinction between 'structural evidence' for operational certification and 'causal evidence' for investment certification is stated but not illustrated with concrete examples or decision criteria that would allow readers to apply the distinction.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which highlight opportunities to strengthen the foundational elements of the Evaluability framework. We address each major comment below, agreeing where revisions are warranted to better support the central claims.

read point-by-point responses

  1. Referee: [Abstract and framework section] Abstract and framework section: The six properties of evaluable evidence are introduced by enumeration without derivation from the Conf(D|E) function or demonstration that they are individually necessary and jointly sufficient to support high-confidence governance decisions. This is load-bearing for the central claim that closing the evaluability gap addresses a category error in current practice.

    Authors: We agree that an explicit derivation would strengthen the argument. In the revised manuscript, we will expand the framework section to derive each of the six properties directly from the requirements of supporting calibrated confidence Conf(D|E) in governance decisions. This will include showing how observability, attributability, intervenability, verifiability, calibration, and temporal validity are individually necessary for evidence to enable high-confidence risk and value decisions, and jointly sufficient to address the category error by shifting focus from system properties to evidentiary capabilities. revision: yes

  2. Referee: [Abstract] Abstract: The assertion that current governance approaches 'pay comparatively little attention to the evidentiary foundations' is presented without supporting measurements, citations to specific frameworks, or case studies showing systematic neglect; this underpins the category-error diagnosis but remains untested within the manuscript.

    Authors: This assertion is grounded in our review of prominent AI governance approaches, which prioritize system properties over evidence-generation mechanisms. To address the concern, the revised version will incorporate specific citations to frameworks such as the NIST AI Risk Management Framework and the EU AI Act, noting their emphasis on properties like safety and fairness with limited explicit treatment of evaluability. As the paper is conceptual rather than empirical, we will not add measurements or case studies but will qualify the statement to reflect this literature analysis, thereby better supporting the category-error diagnosis. revision: partial

Circularity Check

1 steps flagged

Evaluability and its six properties defined by construction to support the governance decisions the framework claims to enable

specific steps
  1. self definitional [Abstract]
    "We introduce Evaluability, defined as the capability of a system to generate, maintain, and renew evidence sufficient to support high-confidence governance decisions over time. We formalize governance decisions as functions of calibrated confidence Conf(D|E) and identify six properties of evaluable evidence: observability, attributability, intervenability, verifiability, calibration, and temporal validity."

    Evaluability is defined in terms of producing evidence 'sufficient to support high-confidence governance decisions,' while the six properties are identified as the properties of such 'evaluable evidence.' The sufficiency claim and the gap-closing role are therefore built into the definition rather than derived from the Conf(D|E) formalization or validated independently, rendering the framework equivalent to its own inputs by construction.

full rationale

The paper's central contribution defines the AI Evaluability Gap as insufficient evidence for high-confidence Conf(D|E) decisions, then introduces Evaluability as the capability to generate evidence sufficient for exactly those decisions, and enumerates the six properties as those of 'evaluable evidence' without deriving them from Conf(D|E) or demonstrating necessity/sufficiency via external validation or case evidence. This makes the claimed missing governance layer self-definitional rather than independently derived. No self-citations or fitted predictions are present; the circularity is purely in the definitional structure of the framework itself.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The framework rests on the domain assumption that current AI governance practice focuses on system properties while neglecting evidence sufficiency; this assumption is asserted rather than measured. No free parameters or invented physical entities appear.

axioms (1)
  • domain assumption AI governance decisions require high-confidence evidence whose sufficiency cannot be taken for granted.
    Stated in the opening paragraph as the shared underlying challenge.
invented entities (1)
  • Evaluability no independent evidence
    purpose: Capability of a system to generate, maintain, and renew evidence sufficient for governance decisions.
    Newly defined construct introduced to address the identified gap.

pith-pipeline@v0.9.1-grok · 5791 in / 1306 out tokens · 13878 ms · 2026-06-26T14:35:59.630142+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 4 canonical work pages

  1. [1]

    Bloomfield, R., and Bishop, P. (2010). Safety and Assurance Cases: Past, Present and Possible Future, an Adelard Perspective. In Making Systems Safer: Proceedings of the Eighteenth Safety-Critical Systems Symposium, 51--67

    2010

  2. [2]

    Brundage, M., Avin, S., Wang, J., et al. (2020). Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. arXiv preprint arXiv:2004.07213

    work page arXiv 2020

  3. [3]

    Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press

    1991

  4. [4]

    Gilboa, I., and Schmeidler, D. (1989). Maxmin Expected Utility with Non-Unique Prior. Journal of Mathematical Economics, 18(2), 141--153

    1989

  5. [5]

    ISO/IEC 42001:2023 -- Information technology -- Artificial intelligence -- Management system

    International Organization for Standardization (2023). ISO/IEC 42001:2023 -- Information technology -- Artificial intelligence -- Management system. International Organization for Standardization, Geneva

    2023

  6. [6]

    Kelly, T., and Weaver, R. (2004). The Goal Structuring Notation -- A Safety Argument Notation. In Proceedings of the Dependable Systems and Networks 2004 Workshop on Assurance Cases

    2004

  7. [7]

    Artificial Intelligence Risk Management Framework (AI RMF 1.0)

    National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, U.S. Department of Commerce

    2023

  8. [8]

    Pearl, J. (2009). Causality: Models, Reasoning, and Inference. 2nd ed. Cambridge University Press

    2009

  9. [9]

    Rubin, D. B. (1974). Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology, 66(5), 688--701

    1974

  10. [10]

    Sah, T., and Srivastava, V. (2026). The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents. arXiv preprint arXiv:2603.19328

    work page arXiv 2026

  11. [11]

    Savage, L. J. (1954). The Foundations of Statistics. John Wiley & Sons

    1954

  12. [12]

    Srivastava, V., et al. (2026). Fundamental Limits of Black-Box Safety Evaluation: Information-Theoretic and Computational Barriers from Latent Context Conditioning. arXiv preprint arXiv:2602.16984

    work page arXiv 2026

  13. [13]

    Srivastava, V., et al. (2026). Quantifying Automation Risk in High-Automation AI Systems: A Bayesian Framework for Failure Propagation and Optimal Oversight. arXiv preprint arXiv:2602.18986

    work page arXiv 2026

  14. [14]

    Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorithmic Learning in a Random World. Springer

    2005