Paper Citation Record · LEDGER

Criteria-Based LLM Relevance Judgments

As of 8 August 2026, this Paper Citation Record lists 38 of 38 outbound references and 5 inbound Pith citation observations for arXiv:2507.09488.

A citation records a reference. It does not transfer a finding from one paper to another.

pith.paper-citation-record.v1
2507.09488 v1

Coverage vector

measured 38 of 38 reference resolution

Typed states for the displayed outbound observations.

Source: paper_references, paper_reference_links, observed 2026-08-06T18:00:21.298350Z

measured 43 of 43 standing notices

One-hop event checks from named stored sources.

Source: scholarly_work_events, retraction_status_cache, observed 2026-08-08T06:32:00.761636+00:00

measured 5 of 5 inbound itemization

Pith citing papers itemized under the disclosed page cap.

Source: paper_references, paper_reference_links, observed 2026-08-03T03:31:57.025464Z

measured 0 of 1 external citation measurements

A source-named dated measurement, never combined with another source.

Source: arxiv_reference, observed 2026-07-03T22:49:00.988765Z

Reference resolution

38 of 38 outbound references displayed

verified exact0
verified fuzzy29
unresolved9
parse uncertain0
malformed identifier0
metadata mismatch0

External citation measurements

No source-named external measurement is stored.

Outbound references

Observation 4e55de38-3c98-46fa-bc43-2ca872946aeb · outbound

This paper cites Can We Use Large Language Models to Fill Relevance Judgment Holes?.

Criteria-Based LLM Relevance Judgments Can We Use Large Language Models to Fill Relevance Judgment Holes?

Reference 1

Resolution: unresolved; no resolver link, observed 2026-08-06T18:00:21.149269Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 48a7e36b-3a97-4b99-a9f7-ab394badd637 · outbound

This paper cites Llms can be fooled into labelling a document as relevant: best café near me; this paper is perfectly relevant.

Criteria-Based LLM Relevance Judgments Llms can be fooled into labelling a document as relevant: best café near me; this paper is perfectly relevant

Reference 2

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.716947Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 3295e0ee-6269-446b-9f02-5754c25466e7 · outbound

This paper cites an unresolved cited work.

Criteria-Based LLM Relevance Judgments Unresolved cited work

Reference 3

Resolution: unresolved; raw_fallback, observed 2026-08-06T18:00:21.706807Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation f386242e-79cf-439c-bbb8-15f8afa5f40d · outbound

This paper cites Barry and Linda Schamber.

Criteria-Based LLM Relevance Judgments Barry and Linda Schamber

Reference 4

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.696511Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 0098a147-bcab-4f39-9460-959af3e8bba2 · outbound

This paper cites The concept of relevance in IR.

Criteria-Based LLM Relevance Judgments The concept of relevance in IR

Reference 5

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.687135Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation d019da2b-9c61-447b-9461-634af7fb4d10 · outbound

This paper cites Beyond topicality: A two stage view of relevance and the retrieval process.

Criteria-Based LLM Relevance Judgments Beyond topicality: A two stage view of relevance and the retrieval process

Reference 6

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.676959Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 8bc23af9-7059-41b9-b8c1-052ce236e32b · outbound

This paper cites an unresolved cited work.

Criteria-Based LLM Relevance Judgments Unresolved cited work

Reference 7

Resolution: unresolved; raw_fallback, observed 2026-08-06T18:00:21.666517Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 6be0c392-5cdf-4471-af07-11c8d05cf50c · outbound

This paper cites Dimensions of Relevance.

Criteria-Based LLM Relevance Judgments Dimensions of Relevance

Reference 8

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.655170Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 92fa95d9-72ee-450b-be99-12f718c485c7 · outbound

This paper cites Overview of the TREC 2020 deep learning track.

Criteria-Based LLM Relevance Judgments Overview of the TREC 2020 deep learning track

Reference 9

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.645451Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 9f9e401e-4b30-46a6-b60e-3e20b6f2f60f · outbound

This paper cites Voorhees.

Criteria-Based LLM Relevance Judgments Voorhees

Reference 10

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.634551Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation a75ccc18-3ed1-44e7-af8d-6d4030693cfd · outbound

This paper cites A workbench for autograding retrieve/generate systems.

Criteria-Based LLM Relevance Judgments A workbench for autograding retrieve/generate systems

Reference 11

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.623238Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 3f42af73-244e-48e9-bae0-16c21eb56037 · outbound

This paper cites Principles and guidelines for the use of llm judges.

Criteria-Based LLM Relevance Judgments Principles and guidelines for the use of llm judges

Reference 12

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.612033Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 143437d1-a5a2-4b60-ae78-a195df2a83d9 · outbound

This paper cites an unresolved cited work.

Criteria-Based LLM Relevance Judgments Unresolved cited work

Reference 13

Resolution: unresolved; no resolver link, observed 2026-08-06T18:00:21.198187Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 42120bb9-7e03-43ae-a938-60ff3c87f7ac · outbound

This paper cites EXAM++: LLM-based Answerability Metrics for IR Evaluation.

Criteria-Based LLM Relevance Judgments EXAM++: LLM-based Answerability Metrics for IR Evaluation

Reference 14

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.593988Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation fbfa5b5a-aef5-45dd-aaa5-a076b269dd02 · outbound

This paper cites Pencils Down! Automatic Rubric-based Eval- uation of Retrieve/Generate Systems.

Criteria-Based LLM Relevance Judgments Pencils Down! Automatic Rubric-based Eval- uation of Retrieve/Generate Systems

Reference 15

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.583974Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 5f6a52a7-0155-4d6e-965c-b703378c7df0 · outbound

This paper cites an unresolved cited work.

Criteria-Based LLM Relevance Judgments Unresolved cited work

Reference 16

Resolution: unresolved; raw_fallback, observed 2026-08-06T18:00:21.574174Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 3bb5820a-542c-438f-9689-0f6013afbe0f · outbound

This paper cites Bruce Croft.

Criteria-Based LLM Relevance Judgments Bruce Croft

Reference 17

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.563843Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 6bdb910c-a3f0-482d-83e5-1f5a1024a4ed · outbound

This paper cites One-shot labeling for automatic relevance estimation.

Criteria-Based LLM Relevance Judgments One-shot labeling for automatic relevance estimation

Reference 18

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.554495Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 5432c96d-3fdb-4476-ab94-ec26a4e5c93e · outbound

This paper cites Crisisfacts: building and evaluating crisis timelines.

Criteria-Based LLM Relevance Judgments Crisisfacts: building and evaluating crisis timelines

Reference 19

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.544892Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 616b0a1e-3378-443d-8267-79795fb19790 · outbound

This paper cites Query performance prediction using relevance judgments generated by large language models.

Criteria-Based LLM Relevance Judgments Query performance prediction using relevance judgments generated by large language models

Reference 20

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.535121Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation a14ee621-5997-4b67-afc8-2df1765166e3 · outbound

This paper cites Golbus, and Javed A.

Criteria-Based LLM Relevance Judgments Golbus, and Javed A

Reference 21

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.523900Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 107a5f42-2112-4f1d-95ea-028290d05d1d · outbound

This paper cites Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework, 2024.

Criteria-Based LLM Relevance Judgments Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework, 2024

Reference 22

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.514146Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 6438008e-e4a7-4f81-92f4-d07e91ad4e6a · outbound

This paper cites Rahmani, Varsha Ramineni, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz.

Criteria-Based LLM Relevance Judgments Rahmani, Varsha Ramineni, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz

Reference 23

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.503471Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 29250cdc-d00d-4f55-9b9f-ddea7d2f741e · outbound

This paper cites Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L.

Criteria-Based LLM Relevance Judgments Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L

Reference 24

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.492219Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation ef6bbbdd-2315-4e54-821a-94cccf3ce142 · outbound

This paper cites Rahmani, Emine Yilmaz, Nick Craswell, and Bhaskar Mitra.

Criteria-Based LLM Relevance Judgments Rahmani, Emine Yilmaz, Nick Craswell, and Bhaskar Mitra

Reference 25

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.481141Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation cce1a809-3c56-4288-8703-0fb9976eb3cf · outbound

This paper cites Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L.

Criteria-Based LLM Relevance Judgments Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L

Reference 26

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.470027Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 538a8fd2-61b5-4b21-a69f-daef79b94928 · outbound

This paper cites Swan: A generic framework for auditing textual conversational systems, 2023.

Criteria-Based LLM Relevance Judgments Swan: A generic framework for auditing textual conversational systems, 2023

Reference 27

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.458551Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 3b85eb03-9b97-4519-9550-4618a42a0a70 · outbound

This paper cites Exam: How to evaluate retrieve-and-generate systems for users who do not (yet) know what they want.

Criteria-Based LLM Relevance Judgments Exam: How to evaluate retrieve-and-generate systems for users who do not (yet) know what they want

Reference 28

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.446632Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 6299d63e-10d7-481f-850f-34288542503b · outbound

This paper cites Relevance: A review of the literature and a framework for thinking on the notion in information science.

Criteria-Based LLM Relevance Judgments Relevance: A review of the literature and a framework for thinking on the notion in information science

Reference 29

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.435613Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation e9881564-ccd6-46d2-baf2-2f35c0fc55fd · outbound

This paper cites Don’t use LLMs to make relevance judgments.Information Retrieval Research Journal, 1(1):10.54195/irrj.19625, March 2025.

Criteria-Based LLM Relevance Judgments Don’t use LLMs to make relevance judgments.Information Retrieval Research Journal, 1(1):10.54195/irrj.19625, March 2025

Reference 30

Resolution: unresolved; no resolver link, observed 2026-08-06T18:00:21.267029Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 5e0823fa-27d5-4f2a-a7f0-513413431ce6 · outbound

This paper cites Is ChatGPT good at search? investigating large language models as re-ranking agents.

Criteria-Based LLM Relevance Judgments Is ChatGPT good at search? investigating large language models as re-ranking agents

Reference 31

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.424555Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 4d61a694-27ef-44c9-810a-e42925a020a8 · outbound

This paper cites Large language models can accurately predict searcher preferences.

Criteria-Based LLM Relevance Judgments Large language models can accurately predict searcher preferences

Reference 32

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.414114Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation c2aefca6-61c7-4556-b164-9a227c056da0 · outbound

This paper cites Together inference api, 2024.

Criteria-Based LLM Relevance Judgments Together inference api, 2024

Reference 33

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.401636Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 90748b70-717d-4419-89ef-5e2d9ec030a8 · outbound

This paper cites Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024.

Criteria-Based LLM Relevance Judgments Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024

Reference 34

Resolution: unresolved; no resolver link, observed 2026-08-06T18:00:21.283508Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 0ce41f22-2051-487c-9a9d-4412b58b58a2 · outbound

This paper cites Le, and Denny Zhou.

Criteria-Based LLM Relevance Judgments Le, and Denny Zhou

Reference 35

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.382904Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation e80025c9-b9b7-4671-9f2d-d34d9e7ef1d8 · outbound

This paper cites Relevance judgment: What do information users consider beyond topicality? Journal of the American Society for Information Science & Technology, 57(7):961–973, May 2006.

Criteria-Based LLM Relevance Judgments Relevance judgment: What do information users consider beyond topicality? Journal of the American Society for Information Science & Technology, 57(7):961–973, May 2006

Reference 36

Resolution: verified fuzzy; raw_fallback, observed 2026-08-06T18:00:21.372243Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 038180a4-47f7-4fad-a042-9ab6179fe786 · outbound

This paper cites an unresolved cited work.

Criteria-Based LLM Relevance Judgments Unresolved cited work

Reference 37

Resolution: unresolved; raw_fallback, observed 2026-08-06T18:00:21.360604Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 7c063baa-5628-4e05-8ca8-3832eb1fd7bf · outbound

This paper cites an unresolved cited work.

Criteria-Based LLM Relevance Judgments Unresolved cited work

Reference 38

Resolution: unresolved; raw_fallback, observed 2026-08-06T18:00:21.349510Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Pith citing papers

Observation cd4fbf59-8fa1-45e0-8e0f-cdaf24cacc6a · inbound

SAGE: Scalable AI Governance & Evaluation cites this paper.

SAGE: Scalable AI Governance & Evaluation Criteria-Based LLM Relevance Judgments

Reference 6

Resolution: unresolved; no resolver link, observed 2026-08-03T03:31:57.025464Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ec6795ed-98e8-4102-9ec4-b83d165037c6 · inbound

mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health cites this paper.

mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health Criteria-Based LLM Relevance Judgments

Reference 2

Resolution: verified exact; arxiv_id, observed 2026-06-30T07:44:21.974095Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 0a0bfecf-24fa-4839-b528-ead159cc4826 · inbound

mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health cites this paper.

mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health Criteria-Based LLM Relevance Judgments

Reference 2

Resolution: verified exact; arxiv_id, observed 2026-07-03T22:49:00.991852Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.

Observation 5c3361bd-f690-4cfc-9754-a09c8b02a848 · inbound

mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health cites this paper.

mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health Criteria-Based LLM Relevance Judgments

Reference 2

Resolution: unresolved; no resolver link, observed 2026-07-12T10:57:57.406511Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation c43ef256-c6fb-4288-9f91-6d797feb36f6 · inbound

Contrastive Reflection for Iterative Prompt Optimization cites this paper.

Contrastive Reflection for Iterative Prompt Optimization Criteria-Based LLM Relevance Judgments

Reference 4

Resolution: verified exact; arxiv_id, observed 2026-07-01T12:25:44.293921Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-08T06:32:00.761636+00:00.