Typed states for the displayed outbound observations.
Source: paper_references, paper_reference_links, observed 2026-08-06T18:00:21.298350Z
Paper Citation Record · LEDGER
Criteria-Based LLM Relevance Judgments
As of 7 August 2026, this Paper Citation Record lists 38 of 38 outbound references and 5 inbound Pith citation observations for arXiv:2507.09488.
A citation records a reference. It does not transfer a finding from one paper to another.
Coverage vector
One-hop event checks from named stored sources.
Source: scholarly_work_events, retraction_status_cache, observed 2026-08-07T06:34:17.273281+00:00
Pith citing papers itemized under the disclosed page cap.
Source: paper_references, paper_reference_links, observed 2026-08-03T03:31:57.025464Z
A source-named dated measurement, never combined with another source.
Source: arxiv_reference, observed 2026-07-03T22:49:00.988765Z
Reference resolution
38 of 38 outbound references displayed
- verified exact0
- verified fuzzy29
- unresolved9
- parse uncertain0
- malformed identifier0
- metadata mismatch0
External citation measurements
No source-named external measurement is stored.
Outbound references
Observation 4e55de38-3c98-46fa-bc43-2ca872946aeb · outbound
This paper cites Can We Use Large Language Models to Fill Relevance Judgment Holes?.
Criteria-Based LLM Relevance Judgments Can We Use Large Language Models to Fill Relevance Judgment Holes?
Reference 1
- Resolution
- unresolved
- no resolver link, observed 2026-08-06T18:00:21.149269Z
Source-reported events for the cited work
Unavailable: canonical work link unavailable.
Observation 48a7e36b-3a97-4b99-a9f7-ab394badd637 · outbound
This paper cites Llms can be fooled into labelling a document as relevant: best café near me; this paper is perfectly relevant.
Criteria-Based LLM Relevance Judgments Llms can be fooled into labelling a document as relevant: best café near me; this paper is perfectly relevant
Reference 2
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.716947Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 3295e0ee-6269-446b-9f02-5754c25466e7 · outbound
This paper cites an unresolved cited work.
Criteria-Based LLM Relevance Judgments Unresolved cited work
Reference 3
- Resolution
- unresolved
- raw_fallback, observed 2026-08-06T18:00:21.706807Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation f386242e-79cf-439c-bbb8-15f8afa5f40d · outbound
This paper cites Barry and Linda Schamber.
Criteria-Based LLM Relevance Judgments Barry and Linda Schamber
Reference 4
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.696511Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 0098a147-bcab-4f39-9460-959af3e8bba2 · outbound
This paper cites The concept of relevance in IR.
Criteria-Based LLM Relevance Judgments The concept of relevance in IR
Reference 5
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.687135Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation d019da2b-9c61-447b-9461-634af7fb4d10 · outbound
This paper cites Beyond topicality: A two stage view of relevance and the retrieval process.
Criteria-Based LLM Relevance Judgments Beyond topicality: A two stage view of relevance and the retrieval process
Reference 6
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.676959Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 8bc23af9-7059-41b9-b8c1-052ce236e32b · outbound
This paper cites an unresolved cited work.
Criteria-Based LLM Relevance Judgments Unresolved cited work
Reference 7
- Resolution
- unresolved
- raw_fallback, observed 2026-08-06T18:00:21.666517Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 6be0c392-5cdf-4471-af07-11c8d05cf50c · outbound
This paper cites Dimensions of Relevance.
Criteria-Based LLM Relevance Judgments Dimensions of Relevance
Reference 8
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.655170Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 92fa95d9-72ee-450b-be99-12f718c485c7 · outbound
This paper cites Overview of the TREC 2020 deep learning track.
Criteria-Based LLM Relevance Judgments Overview of the TREC 2020 deep learning track
Reference 9
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.645451Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 9f9e401e-4b30-46a6-b60e-3e20b6f2f60f · outbound
This paper cites Voorhees.
Reference 10
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.634551Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation a75ccc18-3ed1-44e7-af8d-6d4030693cfd · outbound
This paper cites A workbench for autograding retrieve/generate systems.
Criteria-Based LLM Relevance Judgments A workbench for autograding retrieve/generate systems
Reference 11
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.623238Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 3f42af73-244e-48e9-bae0-16c21eb56037 · outbound
This paper cites Principles and guidelines for the use of llm judges.
Criteria-Based LLM Relevance Judgments Principles and guidelines for the use of llm judges
Reference 12
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.612033Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 143437d1-a5a2-4b60-ae78-a195df2a83d9 · outbound
This paper cites an unresolved cited work.
Criteria-Based LLM Relevance Judgments Unresolved cited work
Reference 13
- Resolution
- unresolved
- no resolver link, observed 2026-08-06T18:00:21.198187Z
Source-reported events for the cited work
Unavailable: canonical work link unavailable.
Observation 42120bb9-7e03-43ae-a938-60ff3c87f7ac · outbound
This paper cites EXAM++: LLM-based Answerability Metrics for IR Evaluation.
Criteria-Based LLM Relevance Judgments EXAM++: LLM-based Answerability Metrics for IR Evaluation
Reference 14
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.593988Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation fbfa5b5a-aef5-45dd-aaa5-a076b269dd02 · outbound
This paper cites Pencils Down! Automatic Rubric-based Eval- uation of Retrieve/Generate Systems.
Criteria-Based LLM Relevance Judgments Pencils Down! Automatic Rubric-based Eval- uation of Retrieve/Generate Systems
Reference 15
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.583974Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 5f6a52a7-0155-4d6e-965c-b703378c7df0 · outbound
This paper cites an unresolved cited work.
Criteria-Based LLM Relevance Judgments Unresolved cited work
Reference 16
- Resolution
- unresolved
- raw_fallback, observed 2026-08-06T18:00:21.574174Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 3bb5820a-542c-438f-9689-0f6013afbe0f · outbound
This paper cites Bruce Croft.
Criteria-Based LLM Relevance Judgments Bruce Croft
Reference 17
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.563843Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 6bdb910c-a3f0-482d-83e5-1f5a1024a4ed · outbound
This paper cites One-shot labeling for automatic relevance estimation.
Criteria-Based LLM Relevance Judgments One-shot labeling for automatic relevance estimation
Reference 18
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.554495Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 5432c96d-3fdb-4476-ab94-ec26a4e5c93e · outbound
This paper cites Crisisfacts: building and evaluating crisis timelines.
Criteria-Based LLM Relevance Judgments Crisisfacts: building and evaluating crisis timelines
Reference 19
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.544892Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 616b0a1e-3378-443d-8267-79795fb19790 · outbound
This paper cites Query performance prediction using relevance judgments generated by large language models.
Criteria-Based LLM Relevance Judgments Query performance prediction using relevance judgments generated by large language models
Reference 20
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.535121Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation a14ee621-5997-4b67-afc8-2df1765166e3 · outbound
This paper cites Golbus, and Javed A.
Criteria-Based LLM Relevance Judgments Golbus, and Javed A
Reference 21
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.523900Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 107a5f42-2112-4f1d-95ea-028290d05d1d · outbound
This paper cites Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework, 2024.
Criteria-Based LLM Relevance Judgments Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework, 2024
Reference 22
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.514146Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 6438008e-e4a7-4f81-92f4-d07e91ad4e6a · outbound
This paper cites Rahmani, Varsha Ramineni, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz.
Criteria-Based LLM Relevance Judgments Rahmani, Varsha Ramineni, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz
Reference 23
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.503471Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 29250cdc-d00d-4f55-9b9f-ddea7d2f741e · outbound
This paper cites Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L.
Criteria-Based LLM Relevance Judgments Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L
Reference 24
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.492219Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation ef6bbbdd-2315-4e54-821a-94cccf3ce142 · outbound
This paper cites Rahmani, Emine Yilmaz, Nick Craswell, and Bhaskar Mitra.
Criteria-Based LLM Relevance Judgments Rahmani, Emine Yilmaz, Nick Craswell, and Bhaskar Mitra
Reference 25
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.481141Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation cce1a809-3c56-4288-8703-0fb9976eb3cf · outbound
This paper cites Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L.
Criteria-Based LLM Relevance Judgments Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L
Reference 26
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.470027Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 538a8fd2-61b5-4b21-a69f-daef79b94928 · outbound
This paper cites Swan: A generic framework for auditing textual conversational systems, 2023.
Criteria-Based LLM Relevance Judgments Swan: A generic framework for auditing textual conversational systems, 2023
Reference 27
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.458551Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 3b85eb03-9b97-4519-9550-4618a42a0a70 · outbound
This paper cites Exam: How to evaluate retrieve-and-generate systems for users who do not (yet) know what they want.
Criteria-Based LLM Relevance Judgments Exam: How to evaluate retrieve-and-generate systems for users who do not (yet) know what they want
Reference 28
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.446632Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 6299d63e-10d7-481f-850f-34288542503b · outbound
This paper cites Relevance: A review of the literature and a framework for thinking on the notion in information science.
Criteria-Based LLM Relevance Judgments Relevance: A review of the literature and a framework for thinking on the notion in information science
Reference 29
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.435613Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation e9881564-ccd6-46d2-baf2-2f35c0fc55fd · outbound
This paper cites Don’t use LLMs to make relevance judgments.Information Retrieval Research Journal, 1(1):10.54195/irrj.19625, March 2025.
Criteria-Based LLM Relevance Judgments Don’t use LLMs to make relevance judgments.Information Retrieval Research Journal, 1(1):10.54195/irrj.19625, March 2025
Reference 30
- Resolution
- unresolved
- no resolver link, observed 2026-08-06T18:00:21.267029Z
Source-reported events for the cited work
Unavailable: canonical work link unavailable.
Observation 5e0823fa-27d5-4f2a-a7f0-513413431ce6 · outbound
This paper cites Is ChatGPT good at search? investigating large language models as re-ranking agents.
Criteria-Based LLM Relevance Judgments Is ChatGPT good at search? investigating large language models as re-ranking agents
Reference 31
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.424555Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 4d61a694-27ef-44c9-810a-e42925a020a8 · outbound
This paper cites Large language models can accurately predict searcher preferences.
Criteria-Based LLM Relevance Judgments Large language models can accurately predict searcher preferences
Reference 32
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.414114Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation c2aefca6-61c7-4556-b164-9a227c056da0 · outbound
This paper cites Together inference api, 2024.
Criteria-Based LLM Relevance Judgments Together inference api, 2024
Reference 33
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.401636Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 90748b70-717d-4419-89ef-5e2d9ec030a8 · outbound
This paper cites Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024.
Criteria-Based LLM Relevance Judgments Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024
Reference 34
- Resolution
- unresolved
- no resolver link, observed 2026-08-06T18:00:21.283508Z
Source-reported events for the cited work
Unavailable: canonical work link unavailable.
Observation 0ce41f22-2051-487c-9a9d-4412b58b58a2 · outbound
This paper cites Le, and Denny Zhou.
Criteria-Based LLM Relevance Judgments Le, and Denny Zhou
Reference 35
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.382904Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation e80025c9-b9b7-4671-9f2d-d34d9e7ef1d8 · outbound
This paper cites Relevance judgment: What do information users consider beyond topicality? Journal of the American Society for Information Science & Technology, 57(7):961–973, May 2006.
Criteria-Based LLM Relevance Judgments Relevance judgment: What do information users consider beyond topicality? Journal of the American Society for Information Science & Technology, 57(7):961–973, May 2006
Reference 36
- Resolution
- verified fuzzy
- raw_fallback, observed 2026-08-06T18:00:21.372243Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 038180a4-47f7-4fad-a042-9ab6179fe786 · outbound
This paper cites an unresolved cited work.
Criteria-Based LLM Relevance Judgments Unresolved cited work
Reference 37
- Resolution
- unresolved
- raw_fallback, observed 2026-08-06T18:00:21.360604Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 7c063baa-5628-4e05-8ca8-3832eb1fd7bf · outbound
This paper cites an unresolved cited work.
Criteria-Based LLM Relevance Judgments Unresolved cited work
Reference 38
- Resolution
- unresolved
- raw_fallback, observed 2026-08-06T18:00:21.349510Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Pith citing papers
Observation cd4fbf59-8fa1-45e0-8e0f-cdaf24cacc6a · inbound
SAGE: Scalable AI Governance & Evaluation cites this paper.
SAGE: Scalable AI Governance & Evaluation Criteria-Based LLM Relevance Judgments
Reference 6
- Resolution
- unresolved
- no resolver link, observed 2026-08-03T03:31:57.025464Z
Source-reported events for the cited work
Unavailable: canonical work link unavailable.
Observation ec6795ed-98e8-4102-9ec4-b83d165037c6 · inbound
mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health cites this paper.
mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health Criteria-Based LLM Relevance Judgments
Reference 2
- Resolution
- verified exact
- arxiv_id, observed 2026-06-30T07:44:21.974095Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 0a0bfecf-24fa-4839-b528-ead159cc4826 · inbound
mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health cites this paper.
mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health Criteria-Based LLM Relevance Judgments
Reference 2
- Resolution
- verified exact
- arxiv_id, observed 2026-07-03T22:49:00.991852Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.
Observation 5c3361bd-f690-4cfc-9754-a09c8b02a848 · inbound
mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health cites this paper.
mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health Criteria-Based LLM Relevance Judgments
Reference 2
- Resolution
- unresolved
- no resolver link, observed 2026-07-12T10:57:57.406511Z
Source-reported events for the cited work
Unavailable: canonical work link unavailable.
Observation c43ef256-c6fb-4288-9f91-6d797feb36f6 · inbound
Contrastive Reflection for Iterative Prompt Optimization cites this paper.
Contrastive Reflection for Iterative Prompt Optimization Criteria-Based LLM Relevance Judgments
Reference 4
- Resolution
- verified exact
- arxiv_id, observed 2026-07-01T12:25:44.293921Z
Source-reported events for the cited work
No event found in the named queried sources as of 2026-08-07T06:34:17.273281+00:00.