Pith. sign in

REVIEW 4 major objections 5 minor 4 references

Measuring Large Language Models Capacity to Annotate Journalistic Sourcing

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new benchmark for LLM annotation of journalistic sourcing finds no model surpasses 42.55% overall accuracy.

desk verdict A useful proof-of-concept benchmark for LLM sourcing annotation, but the headline accuracy numbers are not benchmark-grade: they rest on a single hand-corrected reference set and test-set prompt tuning. read the letter →

arxiv 2501.00164 v2 pith:H5NTROD3 submitted 2024-12-30 cs.CL cs.CY

Subramaniam Vincent , Phoebe Wang , Zhan Shi , Sahas Koka , Yi Fang This is my paper · ORCID
classification cs.CLcs.CY
keywords JournalisticsourcingLargelanguagemodelsLLMbenchmarkingSourceannotationjustificationNewsethicsGroundtruthContentanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that general-purpose large language models cannot yet be trusted to annotate the basic elements of journalistic sourcing, and that this capability deserves its own benchmark. The authors assemble 34 news articles, hand-label 557 sourced statements with five attributes (the sourced text, source type, source name, source title, and the reporter's justification for including the source), and score six LLMs against that ground truth. The headline result is that no model reaches 50% overall accuracy; Gemini Pro 1.5 is best at 42.55%, and source justification matching is the weakest attribute, with every model at 37.25% or below. If this result holds, automated tools that rate the sourcing quality of news would need substantially better models or a different approach, and LLM benchmark victories in law, medicine, or math would not carry over to journalism.

What carries the argument

The carrying mechanism is the five-attribute sourcing annotation schema: Sourced Statement, Type of Source (named person, named organization, document, anonymous source, unnamed group of people), Name of Source, Title of Source, and Source Justification. The prompt is engineered so the model parses the article serially, looking for one source type at a time in a fixed order (anonymous sources, unnamed groups, documents, named persons, named organizations) and emitting JSON data after each pass, following a chain-of-thought prompting style. The scores are produced by matching each model's output to the ground truth with Levenshtein fuzzy matching for names (threshold 0.8) and sentence-embedding semantic matching for statements, titles, and justifications (thresholds 0.8 and 0.55). This combination of schema, serial prompt, and thresholded matcher is what makes the accuracy numbers comparable across models and articles.

What would settle it

Have several journalism-trained annotators independently re-label the same 34 articles with the paper's five-attribute schema, and measure their agreement with the released ground truth on the 557 sourced statements. If agreement is low, or if a re-derived source count differs materially, the reported model rankings and the 42.55% ceiling are artifacts of a single reference set rather than a stable property of the models.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the compound task of journalistic sourcing annotation is currently beyond all six tested LLMs. Using a five-attribute schema inspired by journalism studies and detailed plain-English definitions in the prompt, the best model, Gemini Pro 1.5, achieves 42.55% overall accuracy, meaning it gets the complete five-attribute annotation right less than half the time; DeepSeek R1 follows at 37.19%, Claude 3.5 Sonnet at 32.90%, ChatGPT-4o at 32.28%, Llama 3.1 405B at 30.14%, and Llama 3.1 70B at 25.98%. The components are uneven: models identify source types fairly well (Claude 3.5 Sonnet at 88.49%) and source names and titles near 80%, but sourced statement extraction maxes out at 76.6% (Gemini) and source justification extraction at 37.25% (Gemini). Merging title and justification text improves the score to 51.30% for Claude, still far from usable. The paper concludes that LLMs have more catching up to do and that spotting justifications, the signal that distinguishes bottom-up from authority-driven journalism, is the hardest part.

Load-bearing premise

The load-bearing assumption is that the ground truth annotations are accurate and comprehensive for all 34 articles; since six volunteer graduate students produced them with lead-author corrections and no reported inter-annotator agreement, a noisy or inconsistent reference set would change every model's accuracy numbers.

Editorial extensions

If this is right

  • Automated journalism-ethics annotation is not ready for production: at current accuracy, any tool using these models to audit sourcing would produce many missed sourced statements and incorrect justifications.
  • High scores on conventional LLM benchmarks do not imply competence in journalism annotation, so domain-specific scenarios like this one are needed alongside general-purpose evaluations.
  • Source justification extraction is a separate and harder subproblem than title detection; an ethics auditor cannot treat the two as the same capability.
  • Prompt structure matters: the paper found that serial, one-source-type-at-a-time parsing produced more comprehensive annotations than a single all-at-once instruction, giving benchmark builders a practical lever.
  • Open-source status is not a reliable predictor of annotation quality on this task; Llama 3.1 405B performs close to ChatGPT-4o and Claude 3.5 Sonnet, while the 70B variant lags behind all other models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If low justification scores persist under improved ground truth, then a sourcing-ethics product would likely need a dedicated justification extractor trained on journalism-specific data rather than a general-purpose LLM API.
  • Because the scoring pipeline rewards paraphrase via semantic matching while the prompt asks for exact extraction, the reported justification numbers are sensitive to threshold choice; a stricter exact-word metric might score even lower.
  • Named people and organizations dominate the 34-article sample, so the per-type accuracy scores may overstate capability on anonymous and document sources; a balanced corpus could reveal a wider gap for those types.
  • A natural next test is to add an explicit 'unnamed individual' sixth source type; if models conflate it with anonymous sources, the schema itself, not model size, is the limiting factor.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a scenario, dataset, and evaluation protocol for benchmarking large language models (LLMs) on annotating journalistic sourcing using a five-attribute schema: sourced statement, type of source, name of source, title of source, and source justification. It evaluates six LLMs (Claude 3.5 Sonnet, GPT-4o, Gemini Pro 1.5, Llama 3.1 405B, Llama 3.1 70B, and DeepSeek R1) on 34 news articles with 557 ground-truth sourced statements, using exact, fuzzy, and semantic matching with prescribed thresholds. The central reported finding is that no model reaches 50% overall accuracy on the annotation task, with Gemini Pro 1.5 performing best at 42.55%, and that source justification is the hardest attribute (best match rate 37.25%). The paper releases the dataset, the full prompts, and the model-generated annotations.

Significance. If the accuracy claims were well supported, the paper would contribute a timely and useful benchmark for a relatively neglected but normatively important task: automated annotation of journalistic sourcing for ethics audits. The release of the dataset, prompts, and generated outputs is a genuinely valuable step, and the multi-model comparison across open- and closed-source models is informative. The work is best understood as a proof-of-concept and dataset contribution rather than as a definitive evaluation of LLM capability, because the ground truth is not validated and the evaluation protocol contains methodological gaps (described below). These gaps are fixable, and the paper's positive contributions warrant a major revision rather than rejection.

major comments (4)
  1. [Ground Truth development] The accuracy numbers in Table 6 are computed against a ground truth produced by six volunteer graduate students whose individual annotations were then corrected by the lead author, with no inter-annotator agreement, adjudication protocol, or external validation reported. Because the 'source justification' attribute is defined as any additional characterization explaining why a source is in the story, and the paper itself acknowledges title/justification overlap (Discussion, Figure 9), the reference set is likely to be at least partly idiosyncratic. The reported model scores should therefore be interpreted as agreement with this single reference rather than as task accuracy, and the headline 'below 50%' conclusion could be an artifact of reference-set idiosyncrasy.
  2. [The User Prompt: Logic and learnings] The authors state that prompts were revised iteratively up to version 40 by reviewing what the LLMs caught or missed on the same 34 articles used for the final evaluation. This is test-set contamination: the final scores measure performance on a benchmark whose instructions were tuned to the reference set and the models, and no held-out set was reserved for prompt development. The absolute accuracy figures, including the claim that no model exceeds 50% overall accuracy, are therefore not unbiased estimates of LLM capability on the task as defined.
  3. [Comparison functions and Table 5] The semantic match thresholds (0.8 for sourced statements, 0.55 for titles and justifications) and the fuzzy match threshold (0.8 for names) are ad hoc; no sensitivity analysis or human validation is provided. The central finding that the best justification match rate is only 37.25% depends directly on the 0.55 threshold and on the sentence-level 'highest similarity' comparison, which may be lenient. Without a robustness check or a validation against human judgments of semantic equivalence, the exact numbers in Table 6 are not reproducible in a meaningful sense.
  4. [Accuracy formulae and Overall model accuracy] The paper defines per-attribute match rates in the 'Accuracy formulae' section, but it does not provide an equation for the overall accuracy reported in Table 6 and Figure 10. The verbal definition ('rate of LLMs getting all attributes accurate for each story') is not precise enough to reproduce the reported values, and the overall scores do not appear to be derivable from the per-attribute formulas via any straightforward combination (e.g., product or average). The headline result that no model exceeds 50% overall accuracy therefore lacks a verifiable computation.
minor comments (5)
  1. [Accuracy formulae] The equation for the title match rate appears to be mislabeled as 'Name_matched_num = Title_matched_num / Sentence_matched_num'; this is likely a typographical error that should read 'Title_Match_Rate = Title_matched_num / Sentence_matched_num'.
  2. [Models evaluated] The list of evaluated models contains six models, but the Results section (H2a finding) refers to 'the five LLMs'; the count should be corrected for consistency.
  3. [The User Prompt: Logic and learnings] In bullet 4, 'without or without the language' should read 'with or without the language'.
  4. [References] There are several citation formatting issues, including 'Gottfried at al., 2024' (should be 'et al.') and the fuzzywuzzyR reference, which is given as a package name without a standard author-year citation.
  5. [Dataset description] The full source definitions are provided only in the HuggingFace system prompt file; including the complete schema definitions in an appendix would make the manuscript self-contained and aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracy scores compare LLM outputs against independently produced human ground-truth annotations, and no quantity is defined in terms of the model outputs by construction.

full rationale

The central claim (no evaluated LLM reaches 50% overall accuracy) is an empirical measurement against a human-annotated reference set. The five-attribute schema and its definitions are fixed in the system prompt; the ground-truth files were produced by six volunteer graduate students with lead-author review and correction, independent of the LLM outputs. The accuracy formulae (Statement, Type, Name, Title, and Justification match rates) all compare generated annotations with ground-truth rows; none of the equations reduces to the model outputs by definition or by fitting. The self-citations (Vincent 2023; Vincent et al. 2023; Wang 2024) are motivational or describe prior NLP/annotation work and are not used to derive the accuracy numbers. The iterative prompt revision (versions 1–40) on the same 34 articles and the absence of reported inter-annotator agreement are genuine validity limitations—they affect whether the scores generalize or whether the reference set is reliable—but they are not circularity: the final scores are not statistically forced by the revision procedure and remain contingent empirical outcomes. No uniqueness theorem, ansatz-smuggling, or renamed-known-result pattern is present. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central measurements rest on the authors' chosen evaluation thresholds, on the unvalidated ground truth, and on their five-category schema. No physical or formal entities are invented.

free parameters (3)
  • Semantic match threshold for sourced statements = 0.8
    Chosen by the authors to 'ensure a high level of semantic alignment'; directly determines whether a model's sourced statement counts as matched.
  • Semantic match threshold for titles and justifications = 0.55
    Chosen to 'strike a balance between being overly strict and too lenient'; directly determines title and justification accuracy.
  • Fuzzy match threshold for source names = 80 (on 0-100 scale)
    Chosen to 'balance accuracy and tolerance for minor textual variations'; determines name match accuracy.
assumptions (3)
  • domain assumption The ground truth annotations for the 34 articles are accurate and complete.
    Six volunteer annotators produced annotations later corrected by the lead author, with no inter-annotator agreement measure; all model scores are computed against this reference.
  • domain assumption Sentence-BERT cosine similarity with the chosen thresholds is a valid proxy for human judgment of sourced-statement, title, and justification equivalence.
    No validation against human similarity ratings is reported; the thresholds are set heuristically.
  • domain assumption The five-category schema (sourced statement, type, name, title, justification) captures the salient elements of journalistic sourcing.
    The schema is 'partly inspired' by Gans (2004) but constructed by the authors; the benchmark's validity depends on this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Large Language Models Capacity to Annotate Journalistic Sourcing." pith.science (2026). https://pith.science/paper/H5NTROD3

@misc{pith2026250100164,
  author       = {Pith},
  title        = {Pith review of: Measuring Large Language Models Capacity to Annotate Journalistic Sourcing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H5NTROD3}},
  note         = {Machine review of arXiv:2501.00164}
}
read the original abstract

Since the launch of ChatGPT in late 2022, the capacities of Large Language Models and their evaluation have been in constant discussion and evaluation both in academic research and in the industry. Scenarios and benchmarks have been developed in several areas such as law, medicine and math (Bommasani et al., 2023) and there is continuous evaluation of model variants. One area that has not received sufficient scenario development attention is journalism, and in particular journalistic sourcing and ethics. Journalism is a crucial truth-determination function in democracy (Vincent, 2023), and sourcing is a crucial pillar to all original journalistic output. Evaluating the capacities of LLMs to annotate stories for the different signals of sourcing and how reporters justify them is a crucial scenario that warrants a benchmark approach. It offers potential to build automated systems to contrast more transparent and ethically rigorous forms of journalism with everyday fare. In this paper we lay out a scenario to evaluate LLM performance on identifying and annotating sourcing in news stories on a five-category schema inspired from journalism studies (Gans, 2004). We offer the use case, our dataset and metrics and as the first step towards systematic benchmarking. Our accuracy findings indicate LLM-based approaches have more catching to do in identifying all the sourced statements in a story, and equally, in matching the type of sources. An even harder task is spotting source justifications.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 2 canonical work pages

  1. [1]

    Bommasani, R., Liang, P., & Lee, T. (2023). Holistic evaluation of language models. Annals of the New York Academy of Sciences , 1525 (1), 140–146. https://doi.org/10.1111/nyas.15007 Steensen, S., Belair-Gagnon, V., Graves, L., Kalsnes, B., & Westlund, O. (2022). Journalism and source criticism. Revised Approaches to Assessing Truth-Claims. Journalism Stu...

    work page arXiv 2023

  2. [5]

    https://cplusj2024.github.io/papers/CJ_2024_paper_17.pdf Measuring Large Language Models Capacity to Annotate Journalistic Sourcing Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2024). Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12, 39–57. https:...

    work page Pith review arXiv 2024

  3. [9]

    Shang, X., Peng, Z., Yuan, Q., Khan, S., Xie, L., Fang, Y., & Vincent, S

    [Journal-article]. Shang, X., Peng, Z., Yuan, Q., Khan, S., Xie, L., Fang, Y., & Vincent, S. (2022). DIANES. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval . https://doi.org/10.1145/3477495.3531660 Wang, J., (2024). A Step Towards Automated Ethical Analysis in Journalism: Measuring LLMs’ Per...

    arXiv 2022

  4. [43]

    What's in the News? Towards Identification of Bias by Commission, Omission, and Source Selection (COSS),

    https://scholarcommons.scu.edu/cseng_mstr/43 Full list of annotators . (n.d.). CoreNLP. https://stanfordnlp.github.io/CoreNLP/annotators.html Gottfried, J., Walker, M., (2024, April 14). Most Americans see a place for anonymous sources in news stories, but not all the time. Pew Research Center . Measuring Large Language Models Capacity to Annotate Journal...

    arXiv 2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.