pith. sign in

arxiv: 2606.20963 · v1 · pith:QGTRQDHNnew · submitted 2026-06-18 · 💻 cs.AI · cs.CY· cs.DB

Generative Responsible AI Data Evaluation Schema (GRAIDES) for AI Assurance in Local Government

Ethan Knights , Christopher Conlan , Temilorun Gbolahan , Stephen Waterman , Gurpreet Muctor This is my paper

Pith reviewed 2026-06-26 16:53 UTC · model grok-4.3

classification 💻 cs.AI cs.CYcs.DB
keywords GRAIDESgenerative AIAI evaluationdata schemaAI assurancelocal governmentresponsible AIobservability
0
0 comments X

The pith

GRAIDES treats generative AI evaluations as a data modelling problem to centralize observability and improve consistency across vendors.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces GRAIDES, a lightweight open-source data model designed to structure and centralize evaluation data for generative AI systems used in local government. It supplies practical blueprints for code, architecture, and statistical methods to support benchmarking, tuning, and assurance. A case study from Westminster City Council demonstrates its use in measuring human-model alignment and detecting systematic disagreements between evaluators. By reframing evaluations around data modelling, the approach seeks to reduce fragmentation and enable more reproducible governance activities.

Core claim

GRAIDES is a lightweight open-source data model for centralising AI observability across popular vendors, accompanied by blueprints for code, architecture and statistical evaluation, and illustrated through a case study of Westminster City Council's AI catalogue that measures human-model alignment and detects systematic evaluator disagreement.

What carries the argument

The GRAIDES schema, a data model that structures evaluation records to support observability, benchmarking and assurance for generative AI.

If this is right

  • Enables consistent and comparable benchmarking of generative AI systems across different vendors.
  • Supports detection of systematic disagreements between human evaluators and model outputs.
  • Provides reusable code and architecture patterns for organizational-level AI assurance.
  • Frames evaluation activities as data modelling to make them more reproducible over time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The schema could be extended to track evaluation data over multiple time periods to monitor drift in model performance.
  • Integration with existing municipal data platforms might reduce the effort required for initial adoption.
  • Wider use could surface common patterns of misalignment that apply beyond the local government context.

Load-bearing premise

Organizations will adopt and maintain the GRAIDES schema in a way that produces measurable improvements in AI governance and safety evidence.

What would settle it

An empirical comparison in which organizations using GRAIDES show no measurable gain in evaluation consistency or reproducibility over their prior fragmented approaches.

Figures

Figures reproduced from arXiv: 2606.20963 by Christopher Conlan, Ethan Knights, Gurpreet Muctor, Stephen Waterman, Temilorun Gbolahan.

Figure 1
Figure 1. Figure 1: GRAIDES Specification. Entity Relationship Diagram (ERD) for a source of truth of an organisation’s generative AI activity. 3 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (A) Architectural Blueprint. Data flows from source AI systems (Azure, Google Cloud, Power Platform) into a central storage layer (e.g., a managed Snowflake warehouse or a standalone PostgreSQL container) that hosts GRAIDES components (database tables, dbt pipelines and a simple Python application). (B) Human Grading UI. A frontend presents conversations and configurable grading criteria (integer, boolean,… view at source ↗
Figure 3
Figure 3. Figure 3: Evaluation workflow patterns ranging from simple human approval (A) to more [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Human Approval Results. (A) HR Copilot Agent SME Review. The approval rating percentage is represented for each high-level topic, using 121 golden dataset question and answer pairs. (B) Procurement Assistant Evaluation. Left: SME approvals (100%). Right: Automated embedding-based similarity analysis comparing each generated answer’s cosine sim￾ilarity to the corresponding policy section identified by the S… view at source ↗
Figure 5
Figure 5. Figure 5: Assurance results for a contact centre agent. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Classification model sweep results. (A) Performance heuristics. Overall accuracy and F1‑scores split by positive and negative classes. (B) Prediction rates. Aggregated matri￾ces following standard quadrants. (C) Balanced classification performance. A heatmap of Matthews Correlation Coefficient (MCC) values that balance (B)’s matrices, with a box around the top‑ranking model per theme. (D) Precision-recall … view at source ↗
read the original abstract

Trust in the application of generative Artificial Intelligence (AI) relies on well-governed measurable evidence of performance and safety. In practice, however, evaluation data is often fragmented across systems, inconsistently structured and difficult to compare. We introduce the Generative Responsible AI Data Evaluation Schema (GRAIDES) as a lightweight open-source data model for centralising AI observability across popular vendors. Practical blueprints for code, architecture and statistical evaluation are shared as guidance about how to approach generative system assurance at the organisational level. Illustrative case study results are reported from Westminster City Council's AI catalogue with a focus on measuring human-model alignment including detecting systematic disagreement between evaluators. By framing evaluations as a data modelling problem, GRAIDES provides a practical pathway toward more consistent and reproducible benchmarking, tuning and assurance activities for generative AI systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper introduces the Generative Responsible AI Data Evaluation Schema (GRAIDES), a lightweight open-source data model for centralizing AI observability across vendors. It provides practical blueprints for code, architecture, and statistical evaluation, and reports illustrative case study results from Westminster City Council's AI catalogue focusing on human-model alignment and systematic disagreement detection. The authors claim that framing evaluations as a data modelling problem offers a practical pathway to more consistent and reproducible benchmarking, tuning, and assurance for generative AI systems.

Significance. If adopted, GRAIDES could help address fragmented evaluation data in local government generative AI deployments by providing a shared schema. The open-source release of the schema and code blueprints is a concrete strength that supports reproducibility of the proposed model itself. The case study illustrates application to human-model alignment tasks. However, the claimed pathway to measurable gains in consistency and reproducibility remains untested.

major comments (1)
  1. [Abstract and case study] Abstract and case study section: the claim that GRAIDES 'provides a practical pathway toward more consistent and reproducible benchmarking' is not supported by any comparative metrics, inter-rater reliability deltas, cross-organization variance reductions, or controlled before/after measurements. The Westminster City Council results are described as illustrative only, leaving the central assurance claim without empirical grounding.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their review and agree that the manuscript's central claim about providing a pathway to measurable gains in consistency and reproducibility lacks empirical support. We will revise the abstract and relevant sections to qualify this language and accurately describe the Westminster results as illustrative only.

read point-by-point responses

  1. Referee: [Abstract and case study] Abstract and case study section: the claim that GRAIDES 'provides a practical pathway toward more consistent and reproducible benchmarking' is not supported by any comparative metrics, inter-rater reliability deltas, cross-organization variance reductions, or controlled before/after measurements. The Westminster City Council results are described as illustrative only, leaving the central assurance claim without empirical grounding.

    Authors: We agree with the referee's assessment. The paper presents the Westminster City Council results explicitly as illustrative and does not include comparative metrics, reliability deltas, or controlled measurements to substantiate gains in consistency or reproducibility. In the revised manuscript we will remove the unsubstantiated claim from the abstract and case study section, replacing it with language that positions GRAIDES as a data modelling framework intended to enable such evaluations in future work, without asserting that it has already delivered measurable improvements. revision: yes

Circularity Check

0 steps flagged

No circularity: proposed schema without derivations, fits, or self-referential claims

full rationale

The paper presents GRAIDES as a lightweight open-source data model for centralizing AI observability, along with code blueprints and one illustrative case study from Westminster City Council. No equations, fitted parameters, predictions of derived quantities, or load-bearing self-citations appear in the provided text. The central claim that framing evaluations as a data modelling problem supplies a pathway to consistency is an assertion about the schema's intended utility rather than a result that reduces to its own inputs by construction. The contribution is therefore self-contained as a framework proposal.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms or invented entities are invoked; the work is a data schema proposal.

pith-pipeline@v0.9.1-grok · 5683 in / 827 out tokens · 24950 ms · 2026-06-26T16:53:47.577762+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages

  1. [1]

    (2024, September)

    Ada Lovelace Institute. (2024, September). Buying AI: Is the public sector equipped to procure technology in the public interest? [Discussion paper]. https://www.adalovelaceinstitute.org/report/buying- ai-procurement/ Bowyer, S., Aitchison, L., & Ivanova, D. R. (2025). Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints. arXi...

    work page arXiv 2024

  2. [2]

    Pahune, S., & Akhtar, Z. (2025). Transitioning from MLOps to LLMOps: Navigating the unique challenges of large language models. Information, 16(2),

    2025

  3. [3]

    Palumbo, G., Carneiro, D., & Alves, V. (2024). Objective metrics for ethical AI: a systematic literature review. International Journal of Data Science and Analytics , 20(2), 247-267. Palumbo, G., Guimarães, M., & Carneiro, D. (2025). Observability-driven AI governance: A framework for compliance and audit. In D. H. de la Iglesia, J. F. de Paz Santana, & A...

    work page doi:10.1007/978-3-031-99474-6 2024

  4. [4]

    Siro, C., Aliannejadi, P., & Aliannejadi, M. (2026). Learning to Judge: LLMs Designing and Applying Evaluation Rubrics. arXiv preprint arXiv:2602.08672 . Thakur, A. S., Choudhary, K., Ramayapally, V. S., Vaidyanathan, S., & Hupkes, D. (2025, July). Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. In Proceedings of the Fourth...

    work page arXiv 2026