Paper Citation Record · LEDGER

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits

As of 8 August 2026, this Paper Citation Record lists 22 of 22 outbound references and 0 inbound Pith citation observations for arXiv:2607.02586.

A citation records a reference. It does not transfer a finding from one paper to another.

pith.paper-citation-record.v1
2607.02586 v1

Coverage vector

measured 22 of 22 reference resolution

Typed states for the displayed outbound observations.

Source: paper_references, paper_reference_links, observed 2026-07-12T09:32:32.684011Z

measured 22 of 22 standing notices

One-hop event checks from named stored sources.

Source: scholarly_work_events, retraction_status_cache, observed 2026-08-08T06:32:00.761636+00:00

measured 0 of 0 inbound itemization

Pith citing papers itemized under the disclosed page cap.

Source: paper_references, paper_reference_links

measured 0 of 1 external citation measurements

A source-named dated measurement, never combined with another source.

Source: cited_works

Reference resolution

22 of 22 outbound references displayed

verified exact0
verified fuzzy0
unresolved21
parse uncertain0
malformed identifier1
metadata mismatch0

External citation measurements

No source-named external measurement is stored.

Outbound references

Observation 67b69651-27ea-48de-9559-cbd09bba77c7 · outbound

This paper cites SOK: A Taxonomy of Attack Vectors and Defense Strategies for Agentic Supply Chain Runtime.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits SOK: A Taxonomy of Attack Vectors and Defense Strategies for Agentic Supply Chain Runtime

Reference 1

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ec7d43f1-342d-469a-b184-8565caa45635 · outbound

This paper cites Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries

Reference 2

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 073032d7-3195-4c8d-9845-fbb3ff8f5f94 · outbound

This paper cites FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Models.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Models

Reference 3

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 63e81ac6-31ae-47d2-89b7-09986b98dfaa · outbound

This paper cites Relevant Is Not Warranted: Evidence-Force Calibration for Cited RAG.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Relevant Is Not Warranted: Evidence-Force Calibration for Cited RAG

Reference 4

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 92a4be93-2f68-4136-977b-86b765fe7cb6 · outbound

This paper cites acl-main.442.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits acl-main.442

Reference 5

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 63355ef6-21f7-4fec-a5b4-a837fd0dc16f · outbound

This paper cites When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems

Reference 6

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 7397e43e-ff58-435a-87d0-97a0a157f43c · outbound

This paper cites an unresolved cited work.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Unresolved cited work

Reference 7

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation da547947-b5ee-425b-981b-b5e528965fe1 · outbound

This paper cites an unresolved cited work.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Unresolved cited work

Reference 8

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 3c546353-8c1a-4885-8942-21c177fa27ca · outbound

This paper cites Their numbers illustrate the two failure classes but are not benchmark-validity claims.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Their numbers illustrate the two failure classes but are not benchmark-validity claims

Reference 9

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 2eab1b46-3594-4468-8cc8-589ae7e3d0c3 · outbound

This paper cites Neither scorer is a human-judged label; both cells are reported in thescorer-unvalidatedbucket.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Neither scorer is a human-judged label; both cells are reported in thescorer-unvalidatedbucket

Reference 10

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 2f3bd335-61aa-48f7-9641-35cc0e75bb7a · outbound

This paper cites an unresolved cited work.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Unresolved cited work

Reference 11

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation eab5cc13-b597-47f0-b5a7-945d5def2dae · outbound

This paper cites CrowS- Pairs PLL on paired sentences (Salazar et al., 2020) and BBQ option log-probabilities split by context condition are reference-faithful.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits CrowS- Pairs PLL on paired sentences (Salazar et al., 2020) and BBQ option log-probabilities split by context condition are reference-faithful

Reference 12

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation f4ea4118-cdba-4292-8526-4a1fe9b6cd23 · outbound

This paper cites Canonical.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Canonical

Reference 13

Resolution: malformed identifier; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 9e835470-f028-4bc4-806b-2da5760909f6 · outbound

This paper cites Fix: parametriseformat mcq prompt(label style).

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Fix: parametriseformat mcq prompt(label style)

Reference 14

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation a311968b-d3cc-4405-b991-f0df954d579f · outbound

This paper cites Fix: parametrise the canonical builder.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Fix: parametrise the canonical builder

Reference 15

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation d8cb33fe-e545-4be5-bf4c-16070584cbd3 · outbound

This paper cites Fix: addparaphrase template,synonym substitution.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Fix: addparaphrase template,synonym substitution

Reference 16

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 69efa1dd-0d1d-4f0b-a69c-926a0ff79828 · outbound

This paper cites Fix: canonical option-logprob scoring sets parseability to 1.0; legacy numbers are withdrawn.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Fix: canonical option-logprob scoring sets parseability to 1.0; legacy numbers are withdrawn

Reference 17

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 157fbbe0-8498-4033-ad33-da3116ff6fd0 · outbound

This paper cites Canonical pipeline switches to PLL; CrowS legacy numbers are withdrawn.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Canonical pipeline switches to PLL; CrowS legacy numbers are withdrawn

Reference 18

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 48db66ff-d3c4-421d-9ed7-ecb6e6db5620 · outbound

This paper cites Fix: targeted-token first-token log-probabilities.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Fix: targeted-token first-token log-probabilities

Reference 19

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation c46978f8-b8a5-46f3-b1cf-39b537226363 · outbound

This paper cites Fix: item-level paired bootstrap.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Fix: item-level paired bootstrap

Reference 20

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 52526044-e5cc-47ea-bab1-51b1b4539ba1 · outbound

This paper cites Not fixed; G3 gate flags denominator-dominated cells.

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits Not fixed; G3 gate flags denominator-dominated cells

Reference 21

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 1a944c32-5be6-4918-a54a-e0bd4f7347cf · outbound

This paper cites (A)”, “(B).

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits (A)”, “(B)

Reference 22

Resolution: unresolved; no resolver link, observed 2026-07-12T09:32:32.684011Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Pith citing papers

No inbound Pith citation observations are available.