EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise

Abbas Raza Ali; Moona Zahid; Muhammad Ajmal Siddiqui

REVIEW

EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise

Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid

Artificial Intelligence · Submitted 18 Sep 2026

Pith's verdict: EnterpriseVal provides a well-structured evaluation framework for enterprise GenAI deployments, but its value model's load-bearing parameter ρ_catch is unvalidated, undermining the 'worth scaling' claim.

T0 · AI review

How this review was made: An AI referee (deepseek/deepseek-v4-flash) read the paper on September 21, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

T0 AI review An AI read the paper and answered a fixed list of questions.
T1 Audit Automatic checks that the references and files the paper relies on exist and match.
T2 Compute The paper's code or calculations re-run with the same results.
T3 Formal statement The claims written out in a formal math language a computer can read.
T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 2 major objections · 5 minor · Ask about this paper · The compiled record · Read the full review →

Challenge this review Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2609.21841 (version 2)

Pith ID2QVANDO7, Pith's permanent record of this paper and its review

CitationAbbas Raza Ali, Moona Zahid, Muhammad Ajmal Siddiqui. "EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise." arXiv:2609.21841 (2026). https://pith.science/paper/2QVANDO7

Cite this reviewBibTeX for Pith's review of this paper

Claim paper

Why Pith says this

Highlight any text to comment on it.

Editor's note

read the letter

This paper is worth a careful read if you work on enterprise AI evaluation. It tackles a real problem: the gap between public benchmarks and deployment decisions. The framework is thoughtfully designed—I especially like the formalisation of the socio-technical configuration σ, the conjunctive two-tier gate on conservative confidence bounds, and the insistence that human oversight be measured rather than assumed. The pilot, though small and honestly flagged as incomplete, demonstrates operability in a real bank. The authors separate established results from pilot evidence from hypotheses, which is good practice.

The soft spot is the one the stress-test note and the paper itself flag: ρ_catch. The value-and-risk model (Eq. 10–11) treats the reviewer catch rate as a measured parameter, but no pilot evidence is provided—only a hypothesis that it 'may be below 0.7'. If actual ρ_catch is lower, the annual benefit V_annual could change sign. That makes the 'worth scaling' part of the central claim unsupported for now. The gate framework (Algorithm 2) is not affected, but the full enterprise decision requires that value quantification. So the paper's strongest claim is partially conditional on an unmeasured quantity.

The other weakness is statistical: the pilot lacks confidence intervals, agreement statistics, and repeated runs. The authors acknowledge this and specify the needed experiments. That transparency is commendable, but it means the evidence base is still thin.

Overall, this is a serious, well-reasoned methodology proposal. The framework itself is a genuine contribution, even if the validation is not yet complete. I would send it to peer review—the ideas deserve discussion and the community needs shared infrastructure for this problem. I would not cite it yet for the value model until ρ_catch is empirically grounded, but I might reference the gate design.

Referee report

Significance

If fully validated, EnterpriseVal would address a well-documented failure mode: most enterprise GenAI pilots show no measurable business impact, and agentic projects face high cancellation rates. The framework is notable for (a) formalising the evaluation object as a frozen socio-technical configuration including human oversight, (b) proposing a conjunctive gate on conservative confidence bounds rather than point estimates, and (c) making the reviewer catch rate an explicit parameter in the value model. The pilot, though incomplete, demonstrates operability in a regulated institution. The paper also formulates seven testable hypotheses (H1–H7) that, if supported by future experiments, would substantiate the framework's predictive validity. The main strengths are the formalisation of the evaluation lifecycle and the honest separation of established, documented, proposed, and hypothesis claims throughout the text.

Major objections (2)

[§6, Eq. (10)–(11), §9 H1] The value-and-risk model (Eq. 10) treats the reviewer catch rate ρ_catch as a measured parameter that directly controls the expected error loss π_err = HR·(1 − ρ_catch). The paper itself acknowledges in §9 that ρ_catch is 'the least-studied quantity in the literature' and hypothesises (H1) that it 'may be below 0.7 and decline as trust rises'. No pilot evidence or even a plausible elicitation protocol for ρ_catch is provided. Because V_annual can change sign for ρ_catch values in the hypothesised range (e.g., 0.5 vs. 0.7), the central claim that the system determines whether a workflow is 'worth scaling' is unsupported for the value component. I recommend either (a) reporting a sensitivity analysis over a documented range of ρ_catch values from the literature, or (b) explicitly decoupling the value model from the gate decision and restricting the 'worth scaling' claim to the gate alone until ρ_catch is empirically estimated.
[§7.2, Table 4, §7.4] The pilot's Use Case B results (Table 4) are presented as supporting the framework's operability, yet they violate the framework's own statistical protocol (Algorithm 1, §5). Table 4 reports point estimates without sample sizes (n), confidence intervals, or inter-rater agreement (κ). The gate decision column in Table 4 applies Eq. (9) to point estimates, explicitly cautioned against in §5 ('Gates are taken on the conservative bound') and §7.2 ('without an interval the distinction is meaningless'). The authors note these limitations in §7.4 but then use the table's visual presentation (Figure 5,6) to draw interpretations about model ranking. To substantiate the claim that the framework is 'operable' and produces 'results a governance forum could read', the pilot must report at least n, 95% bootstrap CIs, and κ for every gated metric, with gates re-computed on conservative bounds.

Minor objections (5)

[§3.4, Eq. (3)] Equation (3) reads $E(a,c) = \binom{a+c}{2}$ but the set of possible values is claimed to be {1,...,4}. The binomial coefficient for integer arguments a∈{0..4}, c∈{1..4} yields values {1,3,6,10,15,21,...}, not matching the claimed range. From the context (Figure 3 ) the intended expression appears to be $\lceil (a+c)/2 \rceil$ or perhaps $\min(a+1, c)$ with normalisation. Please correct the equation.
[§7.2, Table 4 footnote †] The footnote states that the irrelevance-rate thresholds are 'inverted (inner should be stricter)'. If thresholds were inverted by mistake, the pilot charter was not correctly set. This undermines the claim that thresholds were 'agreed before results are unblinded' (§6). The issue should be resolved by re-running the charter with correctly oriented thresholds, not merely flagged.
[§6, Algorithm 2] Algorithm 2 checks CI_j against thresholds but does not define what 'CI_j fails θ_out' means when the confidence bound is two-sided. Is it the lower bound for higher-is-better metrics and the upper bound for lower-is-better? Please clarify in the pseudocode.
[§7.3, Use Case C] The efficiency results are based on estimated manual baselines ('practitioner recall') rather than observed timed baselines. The paper states (§4, §7.4) that observed baselines are required for a value claim. The figure captions and abstract should clearly label these as 'estimated' throughout, not bury the caveat in the text.
[§3.2, Eq. (1)] The definition of $g_i$ as 'expert gold artefact' is redundant with the later definition of $G$. The tuple $U$ should either drop $g_i$ from $t_i$ or clarify that $G$ is the set of gold artefacts.

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

EnterpriseVal is a proposed evaluation system that closes the measurement gap between public AI benchmarks and enterprise deployment decisions by combining blinded expert grading with prediction-powered inference, a two-tier threshold…

The reading

The paper argues that most enterprise GenAI pilots fail to show business impact because they are measured on what the model can do, not whether a specific workflow is fit, reliable, safe, and worth scaling on the firm's own data. EnterpriseVal formalizes the evaluation object as a frozen socio-technical configuration including human oversight, defines six metric families (fidelity, utility, efficiency, reliability, assurance, oversight), and scales expert judgment via a calibrated LLM judge with prediction-powered correction. The centerpiece is a two-tier gate that maps metric vectors with confidence bounds to REJECT/CONDITIONAL/SCALE decisions, and a value-and-risk model that explicitly accounts for the reviewer's ability to catch hallucinations (ρ_catch). A pilot in a global bank shows the framework is operable and yields fidelity and efficiency improvements, while acknowledging that the weakest parameter (ρ_catch) lacks empirical support.

What carries the argument

The key machinery is the two-tier gate (Algorithm 2) combined with the value-and-risk model (Eq. 10–11). The gate normalizes each metric relative to outer (minimum viable) and inner (target) thresholds, applies conservative confidence bounds, and outputs a conjunctive REJECT/CONDITIONAL/SCALE decision, preventing one strong metric from masking a weak one. The value model subtracts review costs and expected loss (hallucination rate × (1 − reviewer catch rate ρ_catch)) from time savings, forcing any proposal to shorten review time to account for the risk it creates. The reviewer catch rate ρ_catch is explicitly treated as a measured parameter, though the paper flags it as the least-studied quantity.

Extended reading notes

Core claim

The central claim of EnterpriseVal is that enterprise GenAI deployment failures are fundamentally a measurement problem, not a capability problem. The authors propose that a reliable deployment decision requires evaluating the workflow as a frozen system (model, prompts, retrieval, tools, guardrails, and human oversight), using firm-specific ground truth, blinded expert grading scaled by prediction-powered inference, and conjunctive decision gates on conservative confidence bounds. If the framework is correct, any enterprise can produce an auditable scorecard that either clears the system for scaled deployment with known risk, conditions it on enhanced controls, or rejects it outright, and can estimate the annual value net of review costs and residual risk.

Load-bearing premise

The value model's expected loss term relies on the reviewer catch rate ρ_catch, the probability that a human reviewer detects a hallucinated claim in a fluent draft, which the paper itself identifies as the least-studied quantity and hypothesizes to be below 0.7 and declining with trust.

Editorial extensions

If this is right

If adopted, EnterpriseVal would make enterprise AI scorecards comparable across workflows within a firm, with each use case producing a standardized gate decision and efficacy index.
The conjunctive gate would systematically reject systems that excel on one metric but fail on another, potentially leading to fewer surprise failures in production.
The value model's explicit inclusion of reviewer catch rate would incentivize firms to measure and improve human oversight as part of the AI system, not as an afterthought.
The framework could serve as the operationalization of the EU AI Act's risk-management and human-oversight requirements for high-risk systems.
Prediction-powered inference allows evaluators to use LLM judges at scale without laundering bias into the gate decision, as long as a small human-graded calibration set is maintained.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The dependence on ρ_catch implies that even a flawless AI system (zero hallucination) could still be rejected if the review process is too costly, highlighting the need for firms to invest in reviewer training and ergonomics as part of the deployment.
By requiring frozen configurations and re-evaluation on any change, EnterpriseVal may slow down rapid experimentation in agile contexts, creating a tension between governance and innovation speed.
The conjunctive gate's rejection rate grows with the number of metrics. The paper suggests limiting gated metrics to material harms, but in practice risk committees may want many metrics, potentially leading to excessive CONDITIONAL outcomes that paralyze deployment.

Original abstract

Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically valuable tasks, yet most enterprise GenAI initiatives fail to show a measurable business effect and a large fraction of agentic projects are expected to be cancelled. We argue that this is substantially a measurement problem: public benchmarks answer "what can the model do?", whereas a deployment decision requires "is this workflow fit, reliable, safe and worth scaling - here, on our data, under our controls?". We present EnterpriseVal, a use-case-level evaluation system that closes this gap. It comprises (i) a formal specification of the use case and of the frozen socio-technical configuration under test, model, prompts, retrieval, tools, guardrails and human oversight, with an autonomy level and consequence tier that jointly set the required evaluation intensity; (ii) a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; (iii) a grading protocol that scales blinded expert judgement with calibrated LLM-as-judge scoring through prediction-powered inference; (iv) a two-tier threshold gate, stated as an executable algorithm, that maps metric vectors with confidence bounds to REJECT/CONDITIONAL/SCALE decisions; and (v) a value-and-risk model in which the reviewer catch rate is a measured parameter. We report a pilot across three workflows in a global bank. In credit-memo drafting, human-graded citation precision reached 88% and hallucination rate 1.6% for the best model against gates of 70% and 5%; in procedure transformation, analyst refinement effort fell from an estimated 27.4 to 2.9 hours per document. We separate established results, documented pilot evidence, the proposed system and open hypotheses, and specify the experiments required for full validation

The compiled record

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,902 input tokens.

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.88 · human participants 0.72 · controlled comparison 0.41 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.61 · mentions code or data availability 0.04.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically valuable tasks, yet most enterprise GenAI initiatives fail to show a measurable business effect and a large fraction of agentic projects are expected to be cancelled. We argue that this is substantially a measurement problem: public benchmarks answer "what can the model do?", whereas a deployment decision requires "is this workflow fit, reliable, safe and worth scaling - here, on our data, under our controls?". We present EnterpriseVal, a use-case-level evaluation system that closes this gap. It comprises (i) a formal specification of the use case and of the frozen socio-technical configuration under test, model, prompts, retrieval, tools, guardrails and human oversight, with an autonomy level and consequence tier that jointly set the required evaluation intensity; (ii) a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; (iii) a grading protocol that scales blinded expert judgement with calibrated LLM-as-judge scoring through prediction-powered inference; (iv) a two-tier threshold gate, stated as an executable algorithm, that maps metric vectors with confidence bounds to REJECT/CONDITIONAL/SCALE decisions; and (v) a value-and-risk model in which the reviewer catch rate is a measured parameter. We report a pilot across three workflows in a global bank. In credit-memo drafting, human-graded citation precision reached 88% and hallucination rate 1.6% for the best model against gates of 70% and 5%; in procedure transformation, analyst refinement effort fell from an estimated 27.4 to 2.9 hours per document. We separate established results, documented pilot evidence, the proposed system and open hypotheses, and specify the experiments required for full validation

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically valuable tasks, yet most enterprise GenAI initiatives fail to show a measurable business effect and a large fraction of agentic projects are expected to be cancelled.	background 0.96	0.08	0.10	asserted 1.00
s1	We argue that this is substantially a measurement problem: public benchmarks answer "what can the model do?", whereas a deployment decision requires "is this workflow fit, reliable, safe and worth scaling - here, on our data, under our controls?".	gap 0.66	0.09	0.45	asserted 0.93
s2	We present EnterpriseVal, a use-case-level evaluation system that closes this gap.	contribution 1.00	0.85	0.12	asserted 1.00
s3	It comprises (i) a formal specification of the use case and of the frozen socio-technical configuration under test, model, prompts, retrieval, tools, guardrails and human oversight, with an autonomy level and consequence tier that jointly set the required evaluation intensity; (ii) a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; (iii) a grading protocol that scales blinded expert judgement with calibrated LLM-as-judge scoring through prediction-powered inference; (iv) a two-tier threshold gate, stated as an executable algorithm, that maps metric vectors with confidence bounds to REJECT/CONDITIONAL/SCALE decisions; and (v) a value-and-risk model in which the reviewer catch rate is a measured parameter.	method 0.79	0.82	0.10	asserted 0.99
s4	We report a pilot across three workflows in a global bank.	contribution 0.60	0.94	0.03	asserted 0.98
s5	In credit-memo drafting, human-graded citation precision reached 88% and hallucination rate 1.6% for the best model against gates of 70% and 5%; in procedure transformation, analyst refinement effort fell from an estimated 27.4 to 2.9 hours per document.	result 1.00	0.94	0.11	asserted 1.00
s6	We separate established results, documented pilot evidence, the proposed system and open hypotheses, and specify the experiments required for full validation	future work 0.66	0.76	0.13	asserted 0.97

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

Figures

**Figure 1.** Two gaps between public benchmarks and enterprise decisions. Capability benchmarks measure what a model can do on portable tasks. Enterprises must also establish transfer to their own workflows and ground truth (Gap 1) and convert quality metrics into a value-and-risk case a governance body can act on (Gap 2). EnterpriseVal (eVal) treats evaluation as a systems problem: a repeatable pipeline that takes a frozen workflow configuration and a firm-specific task set and emits an auditable gate decision, an efficacy index and a value estimate with stated uncertainty. It sits between…

**Figure 2.** eVal architecture. Four evaluation layers feed a governance decision; the governance spine consumes artefacts from every layer

**Figure 3.** Evaluation-intensity matrix (Eq. 3). Markers place the pilot use cases: B credit memos (A1,C3), A control assessment (A2,C3), C procedure transformation (A1,C2). families (

**Figure 4.** Two-tier gating. Decisions use the conservative CI bound; an interval straddling both thresholds calls for more samples, not a verdict. with σj = ±1 for higher/lower-is-better, so νj = 0 at θ out , 1 at θ in, negative on failure and capped at νmax (3 in the pilot) so one exceptional metric cannot mask a weak one. With CI− j the conservative bound, Gate(U) =    SCALE ∀j : CI− j ⪰ θ in j CONDITIONAL ∀j : CI− j ⪰ θ out j , ∃j : CI− j ≺ θ in j REJECT otherwise. (9) The rule is conjunctive by design: compensation across metrics is appropriate for ranking models,

**Figure 5.** Use Case B against two-tier thresholds. Dashed/dotted lines mark outer/inner thresholds. Interpretation. Applied mechanically to point estimates, Eq. 9 places Llama 4 in SCALE on all metrics and Gemini 2.5 in CONDITIONAL because its recall of 80% does not strictly exceed 80%. This is exactly what the framework should expose rather than resolve by fiat: without an interval the distinction is meaningless and the correct action is to grade more guidance items (

**Figure 6.** Threshold-normalised scores νj (Use Case B). Origin = θ out, unit circle = θ in, cap 3. Equal-weight and fidelityweighted EI rank the systems differently

**Figure 7.** Use Case C estimated effort by document and pipeline version. and after v2 2.9 h over the seven documents with v2 outputs (−89% against the ten-document manual mean; −88% like-for-like, since the v2 subset’s manual mean is 25.0 h). The v1→v2 improvement is consistent within documents, the more credible signal. Three caveats: the baseline is estimated and clusters at three values (17, 25, 33 h), indicating coarse banding subject to recall bias; the v2 mean is on a subset; and fidelity was not graded, so a faster procedure that silently drops a regulatory step would carry negative va

Figures reproduced from arXiv:2609.21841 by the authors.

References

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.