{"id":"2f12a1da-afdd-428e-8453-16036a769bb3","arxiv_id":"2501.00164","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The best LLM tested, Gemini Pro 1.5, achieves only 42.6% overall accuracy on annotating journalistic sourcing, with source justification extraction the hardest task.","lead":"This paper tests six large language models on labeling who is quoted or cited in news articles, using a new data set of 34 stories. Even the best model, Gemini Pro 1.5, gets only about 43% of the full sourcing annotations right, and all models struggle most with explaining why a source was included.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth reliability is the load-bearing risk: with no inter-annotator agreement reported, every accuracy number is a ratio against a single-adjudicator reference.","rationale":"The reader's weakest assumption — that the ground truth is accurate and comprehensive — is exactly the load-bearing concern. If the human reference is noisy, then the model scores are not measuring a well-defined task; they measure agreement with one particular set of human judgments. This concern is not resolved elsewhere in the paper: the Ground Truth development section describes a training pass and lead-author correction but provides no quantitative reliability evidence, and the schema's boundary cases (e.g., title versus justification) are acknowledged to be ambiguous. I considered whether prompt optimization on the test set is the more damaging issue, but that concern actually cuts in favor of the paper's qualitative conclusion: tuning prompts against the reference can only inflate scores on that reference, so the finding that even tuned models fall below 50% is conservative with respect to generalization. The reference reliability concern is the one that could overturn the central claim, because a different reference could in principle yield materially higher or lower model scores. The concrete test — independent re-annotation of a subset with agreement metrics and score recomputation — directly settles whether the claim is an artifact of the reference. Since the reader already recommended conditional acceptance with reliability reporting, my assessment does not change the verdict; it strengthens the condition.","tokens_in":25652,"tokens_out":6060,"duration_ms":66771,"concrete_test":"Select a random subset of at least 8 of the 34 articles (or all 34 if feasible). Have two independent annotators, trained only with the published system prompt definitions and not involved in the original ground truth, annotate the same articles. Compute per-attribute inter-annotator agreement (Krippendorff's alpha or Cohen's kappa) at the sourced-statement level, then recompute each model's per-attribute and overall scores against each independent reference instead of the released ground truth. Settling criteria: if alpha is at least 0.8 for statement/type/name and at least 0.6 for title/justification, and model scores and model ranking reproduce within roughly 5 points across references, the central claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim — no LLM reaches 50% overall accuracy, with Gemini Pro 1.5 at 42.55% — is computed against a ground truth produced by six volunteer graduate students whose annotations were individually corrected by the lead author (Ground Truth development section). No inter-annotator agreement, adjudication protocol, or dual-coding statistics are reported. This matters because at least two schema attributes are highly subjective: 'source justification' is defined as any characterization explaining why a source is in the story, and the paper itself describes cases where title and justification overlap (Discussion, and the merged-title analysis in Figure 9). If another team of trained annotators would produce materially different reference labels, then the model scores are not estimates of an objective task accuracy; they are agreement rates with one particular reference set. The 'below 50%' conclusion could then largely reflect reference idiosyncrasy rather than LLM incapacity. Iteratively tuning prompts on the same articles (versions 1–40) does not fix this: it can only make scores closer to that same reference, never validate it. The absence of a reliability check is therefore the load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a scenario, dataset, and evaluation protocol for benchmarking large language models (LLMs) on annotating journalistic sourcing using a five-attribute schema: sourced statement, type of source, name of source, title of source, and source justification. It evaluates six LLMs (Claude 3.5 Sonnet, GPT-4o, Gemini Pro 1.5, Llama 3.1 405B, Llama 3.1 70B, and DeepSeek R1) on 34 news articles with 557 ground-truth sourced statements, using exact, fuzzy, and semantic matching with prescribed thresholds. The central reported finding is that no model reaches 50% overall accuracy on the annotation task, with Gemini Pro 1.5 performing best at 42.55%, and that source justification is the hardest attribute (best match rate 37.25%). The paper releases the dataset, the full prompts, and the model-generated annotations.","tokens_in":25868,"tokens_out":6296,"duration_ms":60919,"significance":"If the accuracy claims were well supported, the paper would contribute a timely and useful benchmark for a relatively neglected but normatively important task: automated annotation of journalistic sourcing for ethics audits. The release of the dataset, prompts, and generated outputs is a genuinely valuable step, and the multi-model comparison across open- and closed-source models is informative. The work is best understood as a proof-of-concept and dataset contribution rather than as a definitive evaluation of LLM capability, because the ground truth is not validated and the evaluation protocol contains methodological gaps (described below). These gaps are fixable, and the paper's positive contributions warrant a major revision rather than rejection.","major_comments":[{"comment":"The accuracy numbers in Table 6 are computed against a ground truth produced by six volunteer graduate students whose individual annotations were then corrected by the lead author, with no inter-annotator agreement, adjudication protocol, or external validation reported. Because the 'source justification' attribute is defined as any additional characterization explaining why a source is in the story, and the paper itself acknowledges title/justification overlap (Discussion, Figure 9), the reference set is likely to be at least partly idiosyncratic. The reported model scores should therefore be interpreted as agreement with this single reference rather than as task accuracy, and the headline 'below 50%' conclusion could be an artifact of reference-set idiosyncrasy.","section":"Ground Truth development"},{"comment":"The authors state that prompts were revised iteratively up to version 40 by reviewing what the LLMs caught or missed on the same 34 articles used for the final evaluation. This is test-set contamination: the final scores measure performance on a benchmark whose instructions were tuned to the reference set and the models, and no held-out set was reserved for prompt development. The absolute accuracy figures, including the claim that no model exceeds 50% overall accuracy, are therefore not unbiased estimates of LLM capability on the task as defined.","section":"The User Prompt: Logic and learnings"},{"comment":"The semantic match thresholds (0.8 for sourced statements, 0.55 for titles and justifications) and the fuzzy match threshold (0.8 for names) are ad hoc; no sensitivity analysis or human validation is provided. The central finding that the best justification match rate is only 37.25% depends directly on the 0.55 threshold and on the sentence-level 'highest similarity' comparison, which may be lenient. Without a robustness check or a validation against human judgments of semantic equivalence, the exact numbers in Table 6 are not reproducible in a meaningful sense.","section":"Comparison functions and Table 5"},{"comment":"The paper defines per-attribute match rates in the 'Accuracy formulae' section, but it does not provide an equation for the overall accuracy reported in Table 6 and Figure 10. The verbal definition ('rate of LLMs getting all attributes accurate for each story') is not precise enough to reproduce the reported values, and the overall scores do not appear to be derivable from the per-attribute formulas via any straightforward combination (e.g., product or average). The headline result that no model exceeds 50% overall accuracy therefore lacks a verifiable computation.","section":"Accuracy formulae and Overall model accuracy"}],"minor_comments":[{"comment":"The equation for the title match rate appears to be mislabeled as 'Name_matched_num = Title_matched_num / Sentence_matched_num'; this is likely a typographical error that should read 'Title_Match_Rate = Title_matched_num / Sentence_matched_num'.","section":"Accuracy formulae"},{"comment":"The list of evaluated models contains six models, but the Results section (H2a finding) refers to 'the five LLMs'; the count should be corrected for consistency.","section":"Models evaluated"},{"comment":"In bullet 4, 'without or without the language' should read 'with or without the language'.","section":"The User Prompt: Logic and learnings"},{"comment":"There are several citation formatting issues, including 'Gottfried at al., 2024' (should be 'et al.') and the fuzzywuzzyR reference, which is given as a package name without a standard author-year citation.","section":"References"},{"comment":"The full source definitions are provided only in the HuggingFace system prompt file; including the complete schema definitions in an appendix would make the manuscript self-contained and aid reproducibility.","section":"Dataset description"}],"recommendation":"major_revision","confidential_remarks":"The paper's dataset and task framing are valuable, but the evaluation methodology is not yet at benchmark standard. The most important fixes are (a) reporting inter-annotator agreement and validating the ground truth, and (b) demonstrating that the reported findings are robust to prompt-development contamination and to the ad hoc similarity thresholds. If the authors can add even a small reliability study and a sensitivity analysis, the paper could become acceptable; I do not recommend rejection because the limitations appear fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a genuinely new benchmark scenario for an under-tested task—can general-purpose LLMs annotate journalistic sourcing under a five-attribute schema? The paper ships a public dataset (34 articles, ground truth, prompts, and model outputs) and compares six models with five iterations each. That alone is a real contribution, and the qualitative conclusion that LLMs are poor at extracting source justifications is plausible and probably robust.\n\nBut the quantitative headline—no model reaches 50% overall accuracy, with Gemini Pro 1.5 at 42.55%—is not benchmark-grade yet. The ground truth was produced by six volunteer graduate students with corrections from the lead author, and no inter-annotator agreement or adjudication protocol is reported. For a subjective attribute like source justification, where the paper itself notes title and justification overlap, a different annotation team could easily produce a materially different reference set. The model scores are then agreement rates with one particular reference, not objective task accuracy.\n\nThe second soft spot is test-set contamination. The paper openly describes iterating prompts through version 40 based on model outputs on the same 34 articles used for final evaluation. That doesn't invalidate the qualitative findings, but it means the final numbers are partly fitted to this small sample, and there is no held-out validation. The matching thresholds (0.8 for statements, 0.55 for titles/justifications) are also heuristic, with no sensitivity analysis.\n\nThe paper earns credit for shipping code, prompts, and generated annotations, and for honestly discussing limitations like unbalanced source types and document-sourcing taxonomy. The citation pattern is fine—it engages with Spangher et al. and related work, and self-citations are appropriate.\n\nBottom line: the central claim survives in a weak form—LLMs still have a long way to go on this task, especially on justifications—but the specific scores and model rankings should be treated as provisional. A serious referee should ask for reliability statistics, a held-out prompt-development protocol or explicit reframing, and threshold sensitivity.\n\nFor a computational journalism or LLM-eval audience, this is worth discussing, but I would not cite the numbers as a benchmark yet. Send it to review with a request for revision.","headline":"A useful proof-of-concept benchmark for LLM sourcing annotation, but the headline accuracy numbers are not benchmark-grade: they rest on a single hand-corrected reference set and test-set prompt tuning.","tokens_in":26398,"tokens_out":1720,"would_cite":false,"duration_ms":18580,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark for LLM annotation of journalistic sourcing finds no model surpasses 42.55% overall accuracy.","keywords":["Journalistic sourcing","Large language models","LLM benchmarking","Source annotation","Source justification","News ethics","Ground truth annotation","Content analysis"],"falsifier":"Have several journalism-trained annotators independently re-label the same 34 articles with the paper's five-attribute schema, and measure their agreement with the released ground truth on the 557 sourced statements. If agreement is low, or if a re-derived source count differs materially, the reported model rankings and the 42.55% ceiling are artifacts of a single reference set rather than a stable property of the models.","tokens_in":25460,"feed_emoji":"📰","tokens_out":8013,"duration_ms":80206,"temperature":0.7,"pith_summary":"This paper tries to establish that general-purpose large language models cannot yet be trusted to annotate the basic elements of journalistic sourcing, and that this capability deserves its own benchmark. The authors assemble 34 news articles, hand-label 557 sourced statements with five attributes (the sourced text, source type, source name, source title, and the reporter's justification for including the source), and score six LLMs against that ground truth. The headline result is that no model reaches 50% overall accuracy; Gemini Pro 1.5 is best at 42.55%, and source justification matching is the weakest attribute, with every model at 37.25% or below. If this result holds, automated tools that rate the sourcing quality of news would need substantially better models or a different approach, and LLM benchmark victories in law, medicine, or math would not carry over to journalism.","feed_headline":"No LLM annotates journalistic sourcing with even 50% accuracy","feed_subtitle":"Gemini Pro 1.5 leads the new 34-article sourcing benchmark at 42.55%; spotting why sources are quoted is the weakest task.","key_machinery":"The carrying mechanism is the five-attribute sourcing annotation schema: Sourced Statement, Type of Source (named person, named organization, document, anonymous source, unnamed group of people), Name of Source, Title of Source, and Source Justification. The prompt is engineered so the model parses the article serially, looking for one source type at a time in a fixed order (anonymous sources, unnamed groups, documents, named persons, named organizations) and emitting JSON data after each pass, following a chain-of-thought prompting style. The scores are produced by matching each model's output to the ground truth with Levenshtein fuzzy matching for names (threshold 0.8) and sentence-embedding semantic matching for statements, titles, and justifications (thresholds 0.8 and 0.55). This combination of schema, serial prompt, and thresholded matcher is what makes the accuracy numbers comparable across models and articles.","core_discovery":"On the paper's own terms, the discovery is that the compound task of journalistic sourcing annotation is currently beyond all six tested LLMs. Using a five-attribute schema inspired by journalism studies and detailed plain-English definitions in the prompt, the best model, Gemini Pro 1.5, achieves 42.55% overall accuracy, meaning it gets the complete five-attribute annotation right less than half the time; DeepSeek R1 follows at 37.19%, Claude 3.5 Sonnet at 32.90%, ChatGPT-4o at 32.28%, Llama 3.1 405B at 30.14%, and Llama 3.1 70B at 25.98%. The components are uneven: models identify source types fairly well (Claude 3.5 Sonnet at 88.49%) and source names and titles near 80%, but sourced statement extraction maxes out at 76.6% (Gemini) and source justification extraction at 37.25% (Gemini). Merging title and justification text improves the score to 51.30% for Claude, still far from usable. The paper concludes that LLMs have more catching up to do and that spotting justifications, the signal that distinguishes bottom-up from authority-driven journalism, is the hardest part.","pith_inferences":["If low justification scores persist under improved ground truth, then a sourcing-ethics product would likely need a dedicated justification extractor trained on journalism-specific data rather than a general-purpose LLM API.","Because the scoring pipeline rewards paraphrase via semantic matching while the prompt asks for exact extraction, the reported justification numbers are sensitive to threshold choice; a stricter exact-word metric might score even lower.","Named people and organizations dominate the 34-article sample, so the per-type accuracy scores may overstate capability on anonymous and document sources; a balanced corpus could reveal a wider gap for those types.","A natural next test is to add an explicit 'unnamed individual' sixth source type; if models conflate it with anonymous sources, the schema itself, not model size, is the limiting factor."],"forward_implications":["Automated journalism-ethics annotation is not ready for production: at current accuracy, any tool using these models to audit sourcing would produce many missed sourced statements and incorrect justifications.","High scores on conventional LLM benchmarks do not imply competence in journalism annotation, so domain-specific scenarios like this one are needed alongside general-purpose evaluations.","Source justification extraction is a separate and harder subproblem than title detection; an ethics auditor cannot treat the two as the same capability.","Prompt structure matters: the paper found that serial, one-source-type-at-a-time parsing produced more comprehensive annotations than a single all-at-once instruction, giving benchmark builders a practical lever.","Open-source status is not a reliable predictor of annotation quality on this task; Llama 3.1 405B performs close to ChatGPT-4o and Claude 3.5 Sonnet, while the 70B variant lags behind all other models."],"supporting_citations":[{"why":"Supplies the journalism-studies basis for the five-category sourcing schema used throughout the benchmark.","marker":"Gans, 2004"},{"why":"Defines the scenario-and-benchmark approach the paper adapts to journalism sourcing annotation.","marker":"Bommasani et al., 2023"},{"why":"Closest prior work; tested a fine-tuned GPT-3 for sourced-statement and source-attribution prediction, which this paper extends to instruction-tuned general LLMs.","marker":"Spangher et al, 2023"},{"why":"Provides the sentence-embedding semantic matching used to score sourced statements, titles, and justifications.","marker":"Reimers et al., 2019"},{"why":"Provides the fuzzy string matching used to score source-name accuracy.","marker":"Mouselimis L, 2021"},{"why":"Motivates the serial chain-of-thought style prompt that parses one source type at a time.","marker":"Wei et al., 2022"},{"why":"Supplies the open-source versus closed-source comparison context and fine-tuning suggestion that informs hypothesis H3.","marker":"Alizadeh et al., 2024"}],"fun_headline_variants":["Best LLM scores only 42.55% on journalistic sourcing annotation","Journalistic sourcing annotation: all six LLMs below 50%","Gemini Pro 1.5 leads sourcing benchmark at 42.55% accuracy","LLMs fail to identify source justifications in journalism","Sourcing annotation benchmark: LLMs top out at 42.55%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the ground truth annotations are accurate and comprehensive for all 34 articles; since six volunteer graduate students produced them with lead-author corrections and no reported inter-annotator agreement, a noisy or inconsistent reference set would change every model's accuracy numbers.","fun_headline_variants_meta":{"raw":{"variants":["Best LLM scores only 42.55% on journalistic sourcing annotation","Journalistic sourcing annotation: all six LLMs below 50%","Gemini Pro 1.5 leads sourcing benchmark at 42.55% accuracy","LLMs fail to identify source justifications in journalism","Sourcing annotation benchmark: LLMs top out at 42.55%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":3061,"prompt_tokens":1075,"completion_tokens":1986,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":691,"completion_tokens_details":{"reasoning_tokens":1890}},"tokens_in":691,"tokens_out":1986,"duration_ms":12630,"temperature":1.0,"reasoning_tokens":1890,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:57:24.265609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have several journalism-trained annotators independently re-label the same 34 articles with the paper's five-attribute schema, and measure their agreement with the released ground truth on the 557 sourced statements. If agreement is low, or if a re-derived source count differs materially, the reported model rankings and the 42.55% ceiling are artifacts of a single reference set rather than a stable property of the models.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the scenario-and-benchmark approach the paper adapts to journalism sourcing annotation."}],"review_version":1}