Paper Citation Record · LEDGER

Evaluating the Consistency of LLM Evaluators

As of 23 August 2026, this Paper Citation Record lists 29 of 29 outbound references and 2 inbound Pith citation observations for arXiv:2412.00543.

A citation records a reference. It does not transfer a finding from one paper to another.

pith.paper-citation-record.v1
2412.00543 v1

Coverage vector

measured 29 of 29 reference resolution

Typed states for the displayed outbound observations.

Source: paper_references, paper_reference_links, observed 2026-08-12T05:21:00.661770Z

measured 31 of 31 standing notices

One-hop event checks from named stored sources.

Source: scholarly_work_events, retraction_status_cache, observed 2026-08-22T06:32:14.747728+00:00

measured 2 of 2 inbound itemization

Pith citing papers itemized under the disclosed page cap.

Source: paper_references, paper_reference_links, observed 2026-08-07T05:18:14.014731Z

measured 0 of 1 external citation measurements

A source-named dated measurement, never combined with another source.

Source: pith, observed 2026-08-05T05:30:12.796894Z

Reference resolution

29 of 29 outbound references displayed

verified exact0
verified fuzzy0
unresolved29
parse uncertain0
malformed identifier0
metadata mismatch0

External citation measurements

No source-named external measurement is stored.

Outbound references

Observation f02f586a-ac2c-4705-bc40-fdce5ffbec0c · outbound

This paper cites URL: " 'urlintro :=.

Evaluating the Consistency of LLM Evaluators URL: " 'urlintro :=

Reference 1

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.532973Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ee5cb207-4f47-4d05-8e1e-79dc13cae2e8 · outbound

This paper cites write newline.

Evaluating the Consistency of LLM Evaluators write newline

Reference 2

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.538404Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation e2d3d6cb-6d28-4b96-b101-8a6b36488b08 · outbound

This paper cites Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.

Evaluating the Consistency of LLM Evaluators Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Reference 3

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.542992Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation b4cfbef2-3a7f-4fce-bbfd-8c02a1461542 · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 4

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.547934Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation dec05faf-a604-4c6f-b930-12e9b53f1ec2 · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 5

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.552659Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ecf3fa2e-9581-4994-8317-2ce3947202f3 · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 6

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.556913Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation af05694a-8af7-4bfc-b730-c7464a6afffe · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 7

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.561079Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation fe635d56-dca2-481d-a158-95048e733389 · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 8

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.565731Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 5a0390a3-4065-47a3-9f0e-abf281b2470b · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 9

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.570139Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation e60f3545-b04c-49ab-b407-cbf7bbcf8103 · outbound

This paper cites Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2.

Evaluating the Consistency of LLM Evaluators Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2

Reference 10

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.573930Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation f0a82819-098c-4011-b992-19096d49aa7e · outbound

This paper cites Consistency Analysis of ChatGPT.

Evaluating the Consistency of LLM Evaluators Consistency Analysis of ChatGPT

Reference 11

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.578036Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 88edd7b9-a9b1-49f3-b3d3-e4560d5b9d1a · outbound

This paper cites Mistral 7B.

Evaluating the Consistency of LLM Evaluators Mistral 7B

Reference 12

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.582517Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation e4556369-3056-47f8-a882-86ba43a5707f · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 13

Resolution: unresolved; raw_fallback, observed 2026-08-12T05:21:00.950835Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-22T06:32:14.747728+00:00.

Observation 3f54db1c-3273-49ee-bfa2-2112c8cb5143 · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 14

Resolution: unresolved; raw_fallback, observed 2026-08-12T05:21:00.936603Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-22T06:32:14.747728+00:00.

Observation e9030555-2e1c-4133-a91c-1d9126769faf · outbound

This paper cites Gonzalez, and Ion Stoica.

Evaluating the Consistency of LLM Evaluators Gonzalez, and Ion Stoica

Reference 15

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.594537Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 020c47b3-be67-44a6-bb27-8e5b09342cfd · outbound

This paper cites Hashimoto.

Evaluating the Consistency of LLM Evaluators Hashimoto

Reference 16

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.598539Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation e733e48c-8172-4cf3-9466-d3a42a76bf27 · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 17

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.602406Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 4ff371e4-d1aa-466b-90b9-e91694247319 · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 18

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.606909Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation a1ce2c1b-5821-4a26-9c72-a428877e6450 · outbound

This paper cites Smith, and Yejin Choi.

Evaluating the Consistency of LLM Evaluators Smith, and Yejin Choi

Reference 19

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.610914Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 881ecd2d-e9e1-4738-a270-2507dd34b657 · outbound

This paper cites G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.

Evaluating the Consistency of LLM Evaluators G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

Reference 20

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.619176Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 984fdc1b-27cb-4b9a-9e97-9fd9867253e2 · outbound

This paper cites GPT-4 Technical Report.

Evaluating the Consistency of LLM Evaluators GPT-4 Technical Report

Reference 21

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.624108Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 411e0cb7-329c-431d-aa13-1581c3680244 · outbound

This paper cites Large Language Models are Inconsistent and Biased Evaluators.

Evaluating the Consistency of LLM Evaluators Large Language Models are Inconsistent and Biased Evaluators

Reference 22

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.629509Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 3fa44808-6211-48bd-96e9-8af1af73c2fb · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 23

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.636147Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 26312f29-7c84-4122-8924-7177c04dfb13 · outbound

This paper cites Llama 2: Open Foundation and Fine-Tuned Chat Models.

Evaluating the Consistency of LLM Evaluators Llama 2: Open Foundation and Fine-Tuned Chat Models

Reference 24

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.640487Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation d8f45be9-35c7-4710-8c3a-10a84786261f · outbound

This paper cites Zephyr: Direct Distillation of LM Alignment.

Evaluating the Consistency of LLM Evaluators Zephyr: Direct Distillation of LM Alignment

Reference 25

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.644737Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 4ebe779d-b341-4695-abcb-dd69ba907ab8 · outbound

This paper cites Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.

Evaluating the Consistency of LLM Evaluators Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

Reference 26

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.648924Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ee888c34-c71e-417d-97c1-2737cbc5be2b · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 27

Resolution: unresolved; raw_fallback, observed 2026-08-12T05:21:00.883708Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-22T06:32:14.747728+00:00.

Observation 3209731e-694a-4029-a1b5-2fd6947eca75 · outbound

This paper cites Gonzalez, and Ion Stoica.

Evaluating the Consistency of LLM Evaluators Gonzalez, and Ion Stoica

Reference 28

Resolution: unresolved; no resolver link, observed 2026-08-12T05:21:00.657278Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation bde427b2-f757-4074-b8a5-eb0c4601fff5 · outbound

This paper cites an unresolved cited work.

Evaluating the Consistency of LLM Evaluators Unresolved cited work

Reference 29

Resolution: unresolved; raw_fallback, observed 2026-08-12T05:21:00.862475Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-22T06:32:14.747728+00:00.

Pith citing papers

Observation 7e0a3ad0-2182-4cdc-8996-a3434a27037d · inbound

How Good LLM-Generated Password Policies Are? cites this paper.

How Good LLM-Generated Password Policies Are? Evaluating the Consistency of LLM Evaluators

Reference 8

Resolution: unresolved; no resolver link, observed 2026-08-07T05:18:14.014731Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 8c171ac4-f432-4ce3-9ba7-6edd5f4bac2f · inbound

The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks cites this paper.

The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks Evaluating the Consistency of LLM Evaluators

Reference 18

Resolution: verified exact; local_arxiv, observed 2026-08-05T05:30:12.810095Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-22T06:32:14.747728+00:00.