Paper Citation Record · LEDGER

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education

As of 9 August 2026, this Paper Citation Record lists 51 of 51 outbound references and 0 inbound Pith citation observations for arXiv:2505.18220.

A citation records a reference. It does not transfer a finding from one paper to another.

pith.paper-citation-record.v1
2505.18220 v1

Coverage vector

measured 51 of 51 reference resolution

Typed states for the displayed outbound observations.

Source: paper_references, paper_reference_links, observed 2026-08-07T14:47:14.205177Z

measured 51 of 51 standing notices

One-hop event checks from named stored sources.

Source: scholarly_work_events, retraction_status_cache, observed 2026-08-09T06:31:02.800959+00:00

measured 0 of 0 inbound itemization

Pith citing papers itemized under the disclosed page cap.

Source: paper_references, paper_reference_links

measured 0 of 1 external citation measurements

A source-named dated measurement, never combined with another source.

Source: cited_works

Reference resolution

51 of 51 outbound references displayed

verified exact4
verified fuzzy0
unresolved40
parse uncertain0
malformed identifier2
metadata mismatch5

External citation measurements

No source-named external measurement is stored.

Outbound references

Observation 8c9353ec-e15b-460f-aee7-4e9a92ac957c · outbound

This paper cites Program Synthesis with Large Language Models.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Program Synthesis with Large Language Models

Reference 1

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:09.207018Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 1b2abd35-bf09-4ffc-9f90-40bde7c28e40 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 3

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:47:19.896099Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 8a93f23a-32eb-4163-bb03-817f94a0aefb · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 4

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:09.509601Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 856b6235-c78b-4dd1-9321-475fe62ebc03 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 5

Resolution: verified exact; doi, observed 2026-08-07T14:47:14.930540Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 3b01fb0d-d24d-41bf-b518-74f2f5754560 · outbound

This paper cites what are they not telling me?.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education what are they not telling me?

Reference 6

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:09.737014Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 62e65478-33d9-4e16-bacd-2214384f918d · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 7

Resolution: malformed identifier; raw_fallback, observed 2026-08-07T14:47:18.114168Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 301308e6-8270-4b5f-a048-7bac721c0f3a · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 8

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:09.925135Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 823e9c24-dc71-4b01-b137-69aaf8e22259 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 9

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:10.007298Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 69697e77-5929-45dd-aaa3-71876f5ef916 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 10

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:47:19.619622Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 86ad3969-7263-468c-a15a-2e8de9fbd86b · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 11

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:10.123879Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ee597d55-b3d1-455f-8839-3bf2db6cefad · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 12

Resolution: metadata mismatch; raw_fallback, observed 2026-08-07T14:47:17.532149Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 393c6937-e09f-47a5-ae51-0905c72a0b6a · outbound

This paper cites Yacef, K.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Yacef, K

Reference 13

Resolution: metadata mismatch; raw_fallback, observed 2026-08-07T14:47:17.110472Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 44544e0b-51ea-471c-8589-8f24423cb816 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 14

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:10.445274Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation f6bbea25-d3b7-4af3-9f8e-d079ee9ead81 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 15

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:10.525170Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 96e9b4ce-9367-48de-8e12-2c9a9ec86dc3 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 16

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:10.669599Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ce54a497-4a43-4997-b090-3d9d6ed8ae4d · outbound

This paper cites A Survey on Large Language Models for Code Generation.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education A Survey on Large Language Models for Code Generation

Reference 17

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:10.776573Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation b87d5356-84c4-47a7-9dda-47fff7e56153 · outbound

This paper cites (2024, 2024/10/23).

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education (2024, 2024/10/23)

Reference 18

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:10.875721Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation e21a8f6d-6ded-42db-b898-6e238ffc2c2d · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 19

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:10.964224Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation b9d132b9-cdc5-42ef-b3e9-63151c3a824d · outbound

This paper cites Kasneci, G.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Kasneci, G

Reference 20

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:11.022413Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 215517c4-48bd-49d4-a602-4cb429de8669 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 21

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:11.097483Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 8a8523d4-8838-455f-a214-563360c8d8fe · outbound

This paper cites LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education

Reference 22

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:11.187114Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation fd6d12f9-9869-445a-a1c6-c4acea9229d4 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 23

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:47:19.312063Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation afc336cc-b543-4595-95c7-bb12274d9cd0 · outbound

This paper cites LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs

Reference 24

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:11.386889Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation c90c9834-e830-4663-846f-4bb19d11bf14 · outbound

This paper cites (2024, Oct 11).

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education (2024, Oct 11)

Reference 25

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:11.481778Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation a8374c82-cccb-4e81-9354-d1bb468cea9d · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 26

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:11.580826Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 15eb38b9-14e1-42c8-b6cc-07731fadc32e · outbound

This paper cites Leinonen, J.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Leinonen, J

Reference 27

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:11.713978Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 5e31844d-f055-446a-a5fc-c759ede66c8a · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 28

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:11.847496Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation f9565371-ada2-4256-9551-064b5db66de8 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 29

Resolution: malformed identifier; doi_truncated, observed 2026-08-07T14:47:14.716273Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation a1621cc1-1afb-408c-8d92-281485cccbcf · outbound

This paper cites A Comprehensive Overview of Large Language Models.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education A Comprehensive Overview of Large Language Models

Reference 30

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:12.076169Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 659d69bc-45db-45c0-93a1-cc25ffe933e1 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 31

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:12.175666Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation c20f1547-b53e-4067-b455-d9a1003d0114 · outbound

This paper cites Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models

Reference 32

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:12.277465Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 4eeb331a-c4dd-4025-85df-05af562e678e · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 33

Resolution: metadata mismatch; raw_fallback, observed 2026-08-07T14:47:16.326997Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 8e64514a-0727-47ec-becb-e6c8b3e08072 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 34

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:12.478053Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 1688916a-be06-4d55-bb5c-d30e78a1b640 · outbound

This paper cites It’s Weird That it Knows What I Want.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education It’s Weird That it Knows What I Want

Reference 35

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:12.573037Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation a36453f5-d561-4039-82d8-8a44f64836ce · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 36

Resolution: verified exact; doi, observed 2026-08-07T14:47:14.542649Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 35abebac-2534-47d6-a1e6-7f0d43e7c76b · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 37

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:12.766415Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation f369776f-afe7-482d-b7bd-b2c09cee76c8 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 38

Resolution: metadata mismatch; raw_fallback, observed 2026-08-07T14:47:16.029780Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 9a53eb5d-ef55-44af-aceb-bfd5f53f1a83 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 39

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:47:19.001500Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation ab3d1649-5ff9-4b93-9b00-6a9edc58e755 · outbound

This paper cites Retrieved from https://doi.org/10.1145/3478431.3499397 27 S Kumar, S., Adam Lones, M., Maarek, M., Zantout, H.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Retrieved from https://doi.org/10.1145/3478431.3499397 27 S Kumar, S., Adam Lones, M., Maarek, M., Zantout, H

Reference 40

Resolution: metadata mismatch; raw_fallback, observed 2026-08-07T14:47:15.800322Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation eb85ed24-b182-4b8e-a8f4-7ae06d3f9fe9 · outbound

This paper cites An Analysis of the Automatic Bug Fixing Performance of ChatGPT.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education An Analysis of the Automatic Bug Fixing Performance of ChatGPT

Reference 41

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:13.165320Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 1f86e688-7456-47b0-a88c-aa3537bea92a · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 42

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:13.252809Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation e8fef98d-fd82-49a8-a5f7-5901fff41ac3 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 43

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:13.347216Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 03651d5f-f752-48ae-ab63-a8998f619cdc · outbound

This paper cites (2024, January).

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education (2024, January)

Reference 44

Resolution: verified exact; doi, observed 2026-08-07T14:47:14.361491Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 9eb6d3b5-db31-4af6-8878-07a5ff7fc3c7 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 45

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:13.531590Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation cf86dcae-51f2-403a-8499-c8b9481b4f26 · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 46

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:13.628361Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation ac8f7c39-318b-470b-909f-0699fb8b56fd · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 47

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:13.734371Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 741bbd68-2688-484f-8e64-d3f0c73228ef · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 48

Resolution: verified exact; arxiv_id_nonexistent, observed 2026-08-07T14:47:15.238047Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation 5a3ccd73-4212-444d-8da8-51b7495c92f9 · outbound

This paper cites Repairing Bugs in Python Assignments Using Large Language Models.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Repairing Bugs in Python Assignments Using Large Language Models

Reference 49

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:13.952206Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation a863586f-3f14-4f6a-9e2b-571fb3e178d6 · outbound

This paper cites A Survey of Large Language Models.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education A Survey of Large Language Models

Reference 50

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:14.049257Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Observation 8e5039c6-8418-4bb9-b110-8c0dbe2eeadf · outbound

This paper cites an unresolved cited work.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Unresolved cited work

Reference 51

Resolution: unresolved; raw_fallback, observed 2026-08-07T14:47:18.747436Z

Source-reported events for the cited work

No event found in the named queried sources as of 2026-08-09T06:31:02.800959+00:00.

Observation bd43d10a-19e0-48c3-8609-16e5daa4eea8 · outbound

This paper cites Retrieved from https://doi.org/10.1145/3631802.3631808 29.

Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education Retrieved from https://doi.org/10.1145/3631802.3631808 29

Reference 52

Resolution: unresolved; no resolver link, observed 2026-08-07T14:47:14.205177Z

Source-reported events for the cited work

Unavailable: canonical work link unavailable.

Pith citing papers

No inbound Pith citation observations are available.