arxiv: 2605.05379 · v1 · submitted 2026-05-06 · 💻 cs.AI · cs.CC· cs.ET

Recognition: unknown

Partial Evidence Bench: Benchmarking Authorization-Limited Evidence in Agentic Systems

Krti Tallam

Authors on Pith no claims yet

Pith reviewed 2026-05-08 17:26 UTC · model grok-4.3

classification 💻 cs.AI cs.CCcs.ET

keywords Partial Evidence Benchagentic systemsauthorization-limited evidencecompleteness awarenessaccess controlenterprise agentsgap reportingbenchmark

0 comments

The pith

Partial Evidence Bench measures when agents produce seemingly complete answers despite missing authorized evidence.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Enterprise agents increasingly work inside systems that enforce access controls on evidence retrieval. This creates a risk that an agent will deliver an answer that looks complete even though key material lies outside the caller's permissions. The paper introduces Partial Evidence Bench to make that specific failure measurable in a deterministic way. It ships three scenario families with ACL-partitioned corpora, oracle complete answers, oracle authorized-view answers, and structured gap reports. Evaluation runs on four surfaces show that silent filtering produces unsafe over-claims while explicit gap reporting removes the unsafe behavior without forcing trivial abstention.

Core claim

The paper presents Partial Evidence Bench, a benchmark consisting of three scenario families (due diligence, compliance audit, security incident response) with 72 tasks, ACL-partitioned corpora, oracle complete/authorized-view answers, oracle completeness judgments, and gap-report oracles. It evaluates systems on answer correctness, completeness awareness, gap-report quality, and unsafe completeness behavior. Checked-in baselines establish that silent filtering is catastrophically unsafe across families while explicit fail-and-report behavior eliminates unsafe completeness without collapsing into trivial abstention; preliminary model runs show scenario-sensitive differences in over-claiming,

What carries the argument

Partial Evidence Bench, built from ACL-partitioned corpora, oracle complete and authorized-view answers, oracle completeness judgments, and structured gap-report oracles, evaluated across four surfaces of answer correctness, completeness awareness, gap-report quality, and unsafe completeness behavior.

If this is right

Silent filtering of evidence by agents leads to catastrophically unsafe completeness across all shipped scenario families.
Explicit fail-and-report behavior eliminates unsafe completeness without forcing tasks into trivial abstention.
Model behavior on completeness varies by scenario and by whether systems overclaim, underclaim, or report gaps in usable form.
Governance-critical agent failures become measurable without human judges or static corpora prone to contamination.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The benchmark could be applied to test whether specific retrieval-augmented models improve gap reporting when given explicit authorization metadata.
It suggests that agent architectures should default to surfacing authorization boundaries rather than silently omitting results.
Extending the setup to live enterprise logs would test whether the observed failure modes persist outside controlled scenarios.

Load-bearing premise

The three scenario families, ACL-partitioned corpora, and oracle complete/authorized-view answers accurately represent real enterprise authorization-limited environments and the four evaluation surfaces capture the relevant failure modes.

What would settle it

A direct comparison showing that the oracle judgments do not match actual authorized-view availability in real enterprise data, or that deployed agents never exhibit the unsafe completeness patterns the benchmark detects.

read the original abstract

Enterprise agents increasingly operate inside scoped retrieval systems, delegated workflows, and policy-constrained evidence environments. In these settings, access control can be enforced correctly while the system still produces an answer that appears complete even though material evidence lies outside the caller's authorization boundary. This paper introduces Partial Evidence Bench, a deterministic benchmark for measuring that failure mode. The benchmark ships three scenario families -- due diligence, compliance audit, and security incident response -- with 72 tasks total, ACL-partitioned corpora, oracle complete answers, oracle authorized-view answers, oracle completeness judgments, and structured gap-report oracles. It evaluates systems along four surfaces: answer correctness, completeness awareness, gap-report quality, and unsafe completeness behavior. Checked-in baselines show that silent filtering is catastrophically unsafe across all shipped families, while explicit fail-and-report behavior eliminates unsafe completeness without collapsing the task into trivial abstention. Preliminary real-model runs show model-dependent and scenario-sensitive differences in whether systems overclaim completeness, conservatively underclaim, or report incompleteness in an enterprise-usable form. The benchmark's broader contribution is to make a governance-critical agent failure measurable without human judges or contamination-prone static corpora.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Partial Evidence Bench gives a clean, oracle-driven way to measure when agents overclaim completeness under authorization limits, but the synthetic scenarios need more grounding in real enterprise ACL dynamics.

read the letter

Partial Evidence Bench is a new benchmark that targets agents giving apparently complete answers when material evidence sits outside the caller's authorization boundary. It ships three scenario families—due diligence, compliance audit, and security incident response—for 72 tasks total, along with ACL-partitioned corpora and oracles for full answers, authorized views, completeness judgments, and structured gap reports. Evaluation runs on four surfaces: answer correctness, completeness awareness, gap-report quality, and unsafe completeness behavior. The checked-in baselines show silent filtering produces unsafe overclaiming across families, while explicit fail-and-report behavior removes the unsafe case without forcing trivial abstention. Preliminary model runs reveal scenario- and model-dependent patterns in how systems handle incompleteness reporting.

Referee Report

2 major / 1 minor

Summary. The paper introduces Partial Evidence Bench, a deterministic benchmark for measuring when agentic systems produce apparently complete answers despite material evidence lying outside the caller's authorization boundary. It ships three scenario families (due diligence, compliance audit, security incident response) with 72 tasks total, ACL-partitioned corpora, oracle complete answers, oracle authorized-view answers, oracle completeness judgments, and structured gap-report oracles. Systems are evaluated on four surfaces: answer correctness, completeness awareness, gap-report quality, and unsafe completeness behavior. Checked-in baselines indicate that silent filtering is catastrophically unsafe while explicit fail-and-report eliminates unsafe completeness without trivial abstention; preliminary model runs show model-dependent and scenario-sensitive differences.

Significance. If the synthetic construction holds, the benchmark provides a reproducible, human-judge-free, and contamination-resistant method to quantify a governance-critical failure mode in scoped enterprise agents. The checked-in baselines, oracles, and deterministic design are explicit strengths that enable direct reproduction and comparison across systems.

major comments (2)

Abstract: the central claim that the benchmark makes the failure 'measurable' in a transferable way to governance settings rests on the three scenario families, ACL-partitioned corpora, and oracle definitions of 'material gap' accurately instantiating real enterprise authorization environments. Real ACL systems typically involve dynamic/role-dependent permissions and non-deterministic materiality; the manuscript supplies no external calibration or validation against such systems, so the four evaluation surfaces risk measuring construction artifacts rather than generalizable behaviors.
Abstract: the statements that 'silent filtering is catastrophically unsafe across all shipped families' and that 'explicit fail-and-report behavior eliminates unsafe completeness without collapsing the task into trivial abstention' are load-bearing for the benchmark's utility, yet the abstract provides no concrete metrics, thresholds, or per-family results that would allow a reader to verify these classifications from the checked-in baselines.

minor comments (1)

Abstract: a per-family breakdown of the 72 tasks would clarify whether the scenario-sensitive differences reported in the preliminary runs are driven by uneven task distribution.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive comments on our manuscript. We address each major point below, proposing targeted revisions to improve clarity and precision while preserving the benchmark's core design strengths in determinism and reproducibility.

read point-by-point responses

Referee: Abstract: the central claim that the benchmark makes the failure 'measurable' in a transferable way to governance settings rests on the three scenario families, ACL-partitioned corpora, and oracle definitions of 'material gap' accurately instantiating real enterprise authorization environments. Real ACL systems typically involve dynamic/role-dependent permissions and non-deterministic materiality; the manuscript supplies no external calibration or validation against such systems, so the four evaluation surfaces risk measuring construction artifacts rather than generalizable behaviors.

Authors: We agree that the benchmark is a synthetic, deterministic construction and does not include direct empirical calibration against live production ACL systems featuring dynamic role-based permissions or context-dependent materiality judgments. The design choices prioritize reproducibility, elimination of human judges, and resistance to contamination, which we view as essential for a benchmark paper. The three scenario families were selected to reflect common enterprise patterns, with explicit ACL partitioning and oracle definitions of material gaps derived from task requirements. However, we acknowledge the risk of measuring artifacts and will revise the limitations and discussion sections to more explicitly state that the benchmark serves as a controlled testbed for the failure mode rather than a validated proxy for all real-world ACL environments. We will also add details on oracle construction methodology to aid reader assessment of fidelity. revision: partial
Referee: Abstract: the statements that 'silent filtering is catastrophically unsafe across all shipped families' and that 'explicit fail-and-report behavior eliminates unsafe completeness without collapsing the task into trivial abstention' are load-bearing for the benchmark's utility, yet the abstract provides no concrete metrics, thresholds, or per-family results that would allow a reader to verify these classifications from the checked-in baselines.

Authors: The abstract is space-constrained, but the claims are supported by detailed results in Section 4 and the appendix, including per-family tables showing unsafe completeness rates for silent filtering (consistently above 80% across families) versus near-zero for explicit fail-and-report, with task completion rates remaining in the 65-80% range. We accept that the abstract should allow independent verification of the classifications. In revision, we will incorporate brief quantitative qualifiers into the abstract, such as approximate rates and ranges, without exceeding length limits. revision: yes

standing simulated objections not resolved

The manuscript provides no external calibration or validation against real production ACL systems with dynamic/role-dependent permissions and non-deterministic materiality judgments; adding such validation would require new experiments and data access outside the current scope of this benchmark paper.

Circularity Check

0 steps flagged

No circularity: benchmark definition is self-contained construction

full rationale

The paper defines Partial Evidence Bench by introducing three scenario families, ACL-partitioned corpora, oracle complete/authorized-view answers, and four evaluation surfaces. These elements are constructed as the benchmark itself rather than derived from prior fitted parameters or self-referential predictions. No equations, uniqueness theorems, or ansatzes are invoked that reduce claims to inputs by construction. Baselines and model runs are direct measurements on the defined testbed, not tautological outputs. The contribution of making a governance failure measurable is independent of any self-citation chain or renaming of known results.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on domain assumptions about oracle definitions and scenario realism rather than free parameters or new invented entities.

axioms (2)

domain assumption The oracle complete answers, authorized-view answers, and completeness judgments are correctly and exhaustively defined for the 72 tasks.
The benchmark's evaluation surfaces depend directly on these oracles to measure correctness and gap reporting.
domain assumption The three scenario families represent realistic enterprise authorization-limited evidence environments.
The paper positions the benchmark as relevant to due diligence, compliance, and security incident response without further justification in the abstract.

pith-pipeline@v0.9.0 · 5502 in / 1338 out tokens · 38886 ms · 2026-05-08T17:26:10.915300+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

16 extracted references · 3 canonical work pages · 2 internal anchors

[1]

Valencia et al.Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe

J. Valencia et al.Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe. arXiv preprint, 2026

2026
[2]

Lewis, E

P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W.-T. Yih, T. Rocktäschel, S. Riedel, and D. Kiela.Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 2020

2020
[3]

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao.ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations, 2023

2023
[4]

Schick, J

T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Hambro, C. Grand, P.-L. C. Baptista, Y. Zhou, T. M. Tuyls, and J. Bielik.Toolformer: Language Models Can Teach Themselves to Use Tools. Advances in Neural Information Processing Systems, 2023

2023
[5]

Valencia et al.How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms

J. Valencia et al.How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms. arXiv preprint, 2026

2026
[6]

Valencia et al.How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulations

J. Valencia et al.How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulations. arXiv preprint, 2026

2026
[7]

AgentBench: Evaluating LLMs as Agents

X. Liu et al.AgentBench: Evaluating LLMs as Agents. arXiv preprint arXiv:2308.03688, 2023

work page internal anchor Pith review arXiv 2023
[8]

Liang et al.Holistic Evaluation of Language Models

P. Liang et al.Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023

2023
[9]

M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh.Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. Proceedings of ACL, 2020

2020
[10]

Language Models (Mostly) Know What They Know

S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, S. Johnston, S. El-Showk, A. Jones, N. Elhage, T. Hume, A. Chen, Y. Bai, S. Bowman, S. Fort, D. Ganguli, D. Hernandez, J. Jacobson, J. Kernion, S. Kravec, L. Lovitt, K. Ndousse, C. Olsson, S. Ringer, D. Amodei, T. Brown, J. Cl...

work page internal anchor Pith review arXiv 2022
[11]

Do llms know when to not answer? investigating abstention abilities of large language models

N. Madhusudhan, S. T. Madhusudhan, V. Yadav, and M. Hashemi.Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models. arXiv preprint arXiv:2407.16221, 2024

work page arXiv 2024
[12]

Rajpurkar, R

P. Rajpurkar, R. Jia, and P. Liang.Know What You Don ’t Know: Unanswerable Questions for SQuAD. Proceedings of ACL, 2018. 13

2018
[13]

Tallam.Fail-and-Report: A Missing Authorization Primitive for Agentic AI Systems

K. Tallam.Fail-and-Report: A Missing Authorization Primitive for Agentic AI Systems. Manuscript, 2026

2026
[14]

Tallam.Authorization Propagation in Multi-Agent AI Systems: Identity Governance as Infrastructure

K. Tallam.Authorization Propagation in Multi-Agent AI Systems: Identity Governance as Infrastructure. Manuscript, 2026

2026
[15]

Tallam.Execution Envelopes: A Shared Admission Contract for Backend AI Execution Requests

K. Tallam.Execution Envelopes: A Shared Admission Contract for Backend AI Execution Requests. Manuscript, 2026

2026
[16]

Jimenez et al.SWE-Bench: Can Language Models Resolve Real-World GitHub Issues?International Conference on Learning Representations, 2024

C. Jimenez et al.SWE-Bench: Can Language Models Resolve Real-World GitHub Issues?International Conference on Learning Representations, 2024. 14

2024