Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Knowledge Acquisition on Mass-shooting Events via LLMs for AI-Driven Justice

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4o, prompted with a few labeled examples, is the most effective model for extracting offender, victim, location, and weapon entities from mass-shooting event text, and the paper introduces the first dataset for this task.

desk verdict A niche NER dataset for mass-shooting events with a useful annotation schema, but the LLM comparison is under-specified and one headline claim contradicts its own table. read the letter →

arxiv 2504.12545 v1 pith:LQW7MOI7 submitted 2025-04-17 cs.CY cs.AIcs.CL

Benign John Ihugba , Afsana Nasrin , Ling Wu , Lin Li , Lijun Qian , Xishuang Dong This is my paper · ORCID
classification cs.CYcs.AIcs.CL
keywords namedentityrecognitionmass-shootingeventslargelanguagemodelsfew-shotpromptingAI-drivenjusticeknowledgeacquisitionGPT-4olegalinformationextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to automate knowledge acquisition from mass-shooting reports using named entity recognition, so that investigators and policymakers can quickly get structured facts such as who the offender was, how many victims died, and what weapon was used. Its central claim is that a 41-label annotation schema covering offender, victim, environment, and justice-agency information can be applied to 153 real-world event summaries, and that GPT-4o with few-shot prompting extracts these entities more reliably than GPT-3.5 or o1-mini. If true, this gives AI-driven justice a reusable benchmark and a practical recipe for pulling structured leads from unstructured crisis text.

What carries the argument

The carrying mechanism is few-shot prompting for generative NER, adapted from the GPT-NER approach: the task is reframed from sequence labeling into a text-generation problem where the prompt lists 41 entity types with short labels (ON, OA, CI, TPE, etc.) and supplies a handful of example input–output pairs. The LLM then produces structured entity annotations directly, without fine-tuning, and the annotations are scored with standard Micro Precision, Recall, and F1 metrics. This machinery lets the authors test how much in-context learning alone can transfer a general model to a specialized legal-investigative domain.

What would settle it

Have a second, independent team re-annotate the same 153 event summaries using the same 41-label guidelines, then re-run the three models with the same prompts; if the relative ranking changes or the agreement with the original labels is low, the reported GPT-4o advantage would not survive.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that large language models can perform domain-specific NER in a low-resource setting when given a carefully designed prompt that lists the target entity types and a few annotated examples. The authors report that GPT-4o achieves the highest Micro F1-score (0.6823 with 14 shots), with o1-mini as a close, cheaper alternative, and that increasing the number of shots improves all models, with the largest gains for GPT-4o and o1-mini. They also contribute what they call the first mass-shooting NER dataset, built from public event summaries and annotated under expert supervision with 41 entity subcategories.

Load-bearing premise

The paper assumes the annotations produced by two student labelers under expert supervision are correct and complete enough to serve as ground truth, but it reports no agreement measure between the labelers.

Editorial extensions

If this is right

  • GPT-4o can be used as a drop-in extractor for offender names, victim counts, criminal instruments, and other entities from mass-shooting narratives, reaching about 0.68 Micro F1 with 14 examples.
  • The 41-label schema and the 153-event corpus provide a reusable starting point for building knowledge graphs of mass-shooting events.
  • o1-mini offers a resource-efficient option for simpler NER tasks, with precision close to GPT-4o but lower recall.
  • Adding more in-context examples helps all evaluated models, but the benefit is notably larger for GPT-4o and o1-mini, suggesting these models adapt better to few-shot learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next check is whether the same prompt-and-shot recipe transfers to other crisis domains, such as active-shooter 911 transcripts or court filings; the paper's schema would probably need new labels like 'legal charge' or 'evidence type'.
  • Because the paper does not report inter-annotator agreement, the true ceiling for these models may be lower or higher than the reported F1; an independent reannotation of the same 153 summaries would settle whether the leaderboard is meaningful.
  • The shot-count trend suggests that a small amount of curated, expert-annotated examples could unlock high-quality extraction in other low-resource legal domains, so the main bottleneck is annotation cost, not model capacity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a mass-shooting event NER dataset built from 153 Mother Jones event summaries, with a four-category, 41-subtype annotation schema (Offense, Offender, Victim, Environment/Justice Agency), and evaluates GPT-3.5, GPT-4o, and o1-mini under 5-, 10-, and 14-shot prompting. The reported results in Table III show GPT-4o achieving the highest Micro Precision, Micro Recall, and Micro F1 across prompt sizes, with o1-mini as a competitive resource-efficient alternative. The authors claim this is the first dataset for knowledge acquisition on mass-shooting events using NER and that increasing shot counts improves all models, with larger gains for GPT-4o and o1-mini.

Significance. If the evaluation protocol is made fully transparent and the gold-standard annotations are validated, the released dataset would fill a genuine gap: there is little prior work on structured knowledge acquisition for mass-shooting events, a domain with clear public-safety and legal-investigation relevance. The paper also offers a useful head-to-head comparison of three commercial LLMs on a low-resource, domain-specific NER task, which is of practical value. The schema is thoughtfully organized and the use of a real-world corpus is a strength. However, the current significance is substantially limited by three load-bearing omissions: the train/test or exemplar/evaluation split is not described, the output-parsing and span-matching rule is not specified, and the reliability of the self-annotated gold labels is not quantified. As written, the central ranking in Table III is not independently checkable, and one of the paper's headline claims is contradicted by its own numbers for GPT-3.5.

major comments (4)
  1. [Section V-B, Table III] The paper claims that "increasing the shot count enhances the performance of all models," but this is contradicted by the GPT-3.5 row: Micro F1 falls from 0.4436 at 10 shots to 0.4245 at 14 shots, and Micro Recall falls from 0.3445 to 0.3241. The text later acknowledges only a "marginal improvement" in recall, which itself is not supported by these numbers. This claim appears in the abstract, Section I, Section V-B, and Section VII, so it needs to be corrected or carefully qualified.
  2. [Section IV-B and Section V-B] The evaluation protocol does not specify how the few-shot exemplars are separated from the instances being scored. Section III-A describes a corpus of 153 event summaries, and Section IV-B says each shot illustrates an input snippet paired with the desired output, but no train/test split, cross-validation, or held-out set is described. If the exemplars are drawn from the same 153 summaries that are later scored, the Table III numbers may reflect prompt/test overlap rather than generalization. The authors should specify the split, the exemplar-selection procedure, the number of independent runs, seeds, and whether results are averaged over prompt orderings.
  3. [Section V-A and Section V-B] No output-parsing or span-matching rule is given. The models generate free-form text, but micro precision and micro recall for NER require converting these responses into entity spans and labels and comparing them with gold spans using a specified matching criterion (exact match, partial match, label-only, or other). Without this information, the numbers in Table III cannot be reproduced or independently verified. The authors should describe the parsing step and the exact matching rule used for scoring.
  4. [Section III-B and Figure 3 prompt schema] The gold-label reliability is not demonstrated. Section III-B reports that two student annotators worked under expert supervision but gives no inter-annotator agreement measure (e.g., Cohen's kappa or F1 agreement), and the annotation guidelines are only summarized in Tables I and II. In addition, the prompt schema in Figure 3 lists labels such as OPST, VE, Money, Percent, Art, and Language that do not appear in Tables I-II, so it is unclear what label inventory the models were actually scored against. The authors should report annotator agreement and reconcile the label set between the prompt and the gold-standard schema.
minor comments (5)
  1. [Section VI-B] Several quantitative claims in Related Work are stated without specific supporting citations, including "increase recall by approximately 5%," "F1-scores exceeding 93%," and a "10% reduction in entity disambiguation errors." These need explicit references or should be removed.
  2. [Figure 4] The per-entity F1 comparison in Figure 4 is difficult to read for low-frequency labels, and the text makes claims such as "up to 4-times higher F1-scores" without reporting the underlying values. A supplementary table with numerical F1 per entity type would make the claims checkable.
  3. [Section III-B and Section IV-B] Section III-B says annotations follow the standard BIO format, but the prompt in Figure 3 asks for generative outputs listing key entities. The relationship between the BIO-annotated gold data and the generative few-shot format should be clarified.
  4. [References] Some references appear mismatched to the claims they support; for example, reference [24] is cited for the claim that NLP models achieve over 90% accuracy in extracting case elements from court transcripts, but [24] is the GPT-3 paper and does not appear to contain that result. Please verify all forward references.
  5. [Throughout] The manuscript has minor typographical and style inconsistencies, such as inconsistent hyphenation of "mass shooting" versus "mass-shooting," inconsistent capitalization of "Micro recall" versus "Micro Recall," and occasional tense inconsistencies in the abstract and introduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical LLM comparison scored against human-annotated labels, with no fitted parameter, definitional identity, or load-bearing self-citation chain.

full rationale

This paper is an empirical benchmark study rather than a derivation, so the enumerated circularity patterns do not apply. The central claim, that GPT-4o outperforms GPT-3.5 and o1-mini on mass-shooting NER, is supported by the reported Micro Precision, Micro Recall, and Micro F1 scores in Table III, which are computed by comparing model outputs with human-annotated labels produced by two student annotators under expert supervision through iterative labeling, cross-verification, and adjudication (Section III-B). No parameter is fitted to a subset of the evaluation data and then renamed as a prediction; no formula defines the reported metric in terms of the model outputs plus a fitted constant; and no load-bearing self-citation is used to justify the ranking. The dataset is self-constructed and annotated with the authors' own guidelines, which is standard practice in new-domain NER evaluation and is not circular by itself. Concerns such as the absence of an explicit train/test split, the lack of an inter-annotator agreement measure, and the undocumented output-parsing rule are experimental-validity and reproducibility issues, not circular reductions. Under the hard rule requiring a quoted equation or fitted parameter that reduces a claimed result to its own inputs, no circular step can be exhibited, so the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper rests on the quality of the Mother Jones corpus and the reliability of its own annotation guidelines. No free parameters are fitted, and no new theoretical entities are introduced.

assumptions (2)
  • domain assumption Mother Jones mass-shooting dataset is a sufficiently representative source of event summaries for NER.
    Section III-A states the data was collected from the Mother Jones dataset without justifying its coverage or representativeness.
  • domain assumption Human annotations produced by two trained students under expert supervision are accurate ground truth.
    Section III-B describes the annotation process but reports no inter-annotator agreement or validation against an external standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Knowledge Acquisition on Mass-shooting Events via LLMs for AI-Driven Justice." pith.science (2026). https://pith.science/paper/LQW7MOI7

@misc{pith2026250412545,
  author       = {Pith},
  title        = {Pith review of: Knowledge Acquisition on Mass-shooting Events via LLMs for AI-Driven Justice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LQW7MOI7}},
  note         = {Machine review of arXiv:2504.12545}
}
read the original abstract

Mass-shooting events pose a significant challenge to public safety, generating large volumes of unstructured textual data that hinder effective investigations and the formulation of public policy. Despite the urgency, few prior studies have effectively automated the extraction of key information from these events to support legal and investigative efforts. This paper presented the first dataset designed for knowledge acquisition on mass-shooting events through the application of named entity recognition (NER) techniques. It focuses on identifying key entities such as offenders, victims, locations, and criminal instruments, that are vital for legal and investigative purposes. The NER process is powered by Large Language Models (LLMs) using few-shot prompting, facilitating the efficient extraction and organization of critical information from diverse sources, including news articles, police reports, and social media. Experimental results on real-world mass-shooting corpora demonstrate that GPT-4o is the most effective model for mass-shooting NER, achieving the highest Micro Precision, Micro Recall, and Micro F1-scores. Meanwhile, o1-mini delivers competitive performance, making it a resource-efficient alternative for less complex NER tasks. It is also observed that increasing the shot count enhances the performance of all models, but the gains are more substantial for GPT-4o and o1-mini, highlighting their superior adaptability to few-shot learning scenarios.

Figures

Figures reproduced from arXiv: 2504.12545 by the authors.

Figure 1
Figure 1. An example of NER on a mass-shooting event for knowledge acquisition. It includes various types of named [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distributions of named entities involved in mass-shooting events. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Prompt schema of NER on mass-shooting events. of the o1 model series, specifically designed to enhance reasoning capabilities in tasks such as math, coding, and cybersecurity. It oper￾ates with reduced resource requirements while maintaining strong performance. Previous stud￾ies indicate that while o1-mini may not fully match the depth of understanding of the larger o1 model in highly intricate general con￾texts, it… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance comparison regarding F1-score across different types of named entities with 14 shots. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Named-Entity Recognition in the Crime Domain (CrimeNER): Case Study and Dataset

    cs.CL 2026-03 conditional novelty 6.0 of 10

    CrimeNER-db is a new, publicly released 1,568-document manually annotated corpus for crime-domain NER with a coarse/fine label hierarchy and zero-/few-shot benchmark results.

Reference graph

Works this paper leans on

32 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Advancing Legal Reasoning: The Integration of AI to Navigate Complexities and Biases in Global Jurisprudence with Semi-Automated Arbitration Processes (SAAPs)

    M. De’Shazer, “Advancing legal reasoning: The integration of ai to navigate complexities and biases in global ju- risprudence with semi-automated arbitration processes (saaps),” arXiv preprint arXiv:2402.04140, 2024

    work page Pith review arXiv 2024

  2. [2]

    Artificial intelligence use in criminal investigations gains traction,

    A. R. Crime, “Artificial intelligence use in criminal investigations gains traction,” 2024. [Online]. Available: https://www.aetv.com/real-crime/ artificial-intelligence-use-in-criminal-investigations

    work page 2024

  3. [3]

    A survey on knowledge graphs: Representation, acquisition, and appli- cations,

    S. Ji, S. Pan, E. Cambria, P. Marttinen, and P. S. Yu, “A survey on knowledge graphs: Representation, acquisition, and appli- cations,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 2, 2022, pp. 494–514

    work page 2022

  4. [4]

    From data to knowledge: Building a single source of truth,

    GraphAware, “From data to knowledge: Building a single source of truth,” 2023. [Online]. Available: https://graphaware. com/blog/data-to-knowledge/

    work page 2023

  5. [5]

    Gpt-ner: Named entity recognition via large language models,

    S. Wang, X. Sun, X. Li, R. Ouyang, F. Wu, T. Zhang, J. Li, and G. Wang, “Gpt-ner: Named entity recognition via large language models,” arXiv preprint arXiv:2304.10428, 2023

    arXiv 2023

  6. [6]

    Knowledge graph enhanced neural machine translation via multi-task learning on sub-entity granularity,

    Y . Zhao, L. Xiang, J. Zhu, J. Zhang, Y . Zhou, and C. Zong, “Knowledge graph enhanced neural machine translation via multi-task learning on sub-entity granularity,” in Proceedings of the 28th International Conference on Computational Linguistics, D. Scott, N. Bel, and C. Zong, Eds. Barcelona, Spain (Online): International Committee on Computational Lingu...

    work page 2020

  7. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  8. [8]

    Study highlights need for public health approach in news coverage of gun violence,

    The Journalist’s Resource, “Study highlights need for public health approach in news coverage of gun violence,”

    work page

Show all 32 references
  1. [9]

    Named entity recognition: A literature survey,

    R. Sharnagat, “Named entity recognition: A literature survey,” Center For Indian Language Technology, 2014, pp. 1–27

    2014

  2. [10]

    A survey on prompting techniques in llms,

    P. Bhandari, “A survey on prompting techniques in llms,” arXiv preprint arXiv:2312.03740, 2023

    2023 arXiv

  3. [11]

    Evaluating the logical reasoning ability of chatgpt and gpt-4,

    H. Liu, R. Ning, Z. Teng, J. Liu, Q. Zhou, and Y . Zhang, “Evaluating the logical reasoning ability of chatgpt and gpt-4,” arXiv preprint arXiv:2304.03439, 2023

    2023 arXiv

  4. [12]

    Chatgpt-3.5: An overview and limitations,

    blocshop, “Chatgpt-3.5: An overview and limitations,”

  5. [13]

    Chatgpt-4o vs o1-mini: A comparison,

    A. API, “Chatgpt-4o vs o1-mini: A comparison,”

  6. [14]

    Openai’s new o1 and o1-mini models: What you need to know,

    Turtleverse, “Openai’s new o1 and o1-mini models: What you need to know,” 2025, accessed: 2025- 01-29. [Online]. Available: https://turtleverse.com/ openais-new-o1-and-o1-mini-models-what-you-need-to-know/

    2025

  7. [15]

    Gpt-4 — Wikipedia, the free ency- clopedia,

    Wikipedia contributors, “Gpt-4 — Wikipedia, the free ency- clopedia,” https://en.wikipedia.org/w/index.php?title=GPT-4& oldid=1273223835, 2025, [Online; accessed 4-February-2025]

    2025

  8. [16]

    Successive prompting for decomposing complex questions,

    D. Dua, S. Gupta, S. Singh, and M. Gardner, “Successive prompting for decomposing complex questions,” arXiv preprint arXiv:2212.04092, 2022

    2022 arXiv

  9. [17]

    A survey on efficient training of transformers,

    B. Zhuang, J. Liu, Z. Pan, H. He, Y . Weng, and C. Shen, “A survey on efficient training of transformers,” arXiv preprint arXiv:2302.01107, 2023

    2023 arXiv

  10. [18]

    Mobilebert: Task-agnostic compression of bert by progressive knowledge transfer,

    Z. Sun, H. Yu, X. Song, R. Liu, Y . Yang, and D. Zhou, “Mobilebert: Task-agnostic compression of bert by progressive knowledge transfer,” 2019

    2019

  11. [19]

    Making pre-trained language models better few-shot learners,

    T. Gao, A. Fisch, and D. Chen, “Making pre-trained language models better few-shot learners,” arXiv preprint arXiv:2012.15723, 2020

    2012 arXiv

  12. [20]

    Prompt engi- neering a prompt engineer,

    Q. Ye, M. Axmed, R. Pryzant, and F. Khani, “Prompt engi- neering a prompt engineer,” arXiv preprint arXiv:2311.05661, 2023

    2023 arXiv

  13. [21]

    A survey on deep learning for named entity recognition,

    J. Li, A. Sun, J. Han, and C. Li, “A survey on deep learning for named entity recognition,” IEEE transactions on knowledge and data engineering, vol. 34, no. 1, 2020, pp. 50–70

    2020

  14. [22]

    Staying in control of technology: predictive policing, democracy, and digital sovereignty,

    M. Leese, “Staying in control of technology: predictive policing, democracy, and digital sovereignty,” Democratization, vol. 31, no. 5, 2024, pp. 963–978. [Online]. Available: https://doi.org/10.1080/13510347.2023.2197217

    2024

  15. [23]

    Legal case classification using machine learning with nlp,

    M. S. Tomar and V . Gupta, “Legal case classification using machine learning with nlp,” in 2023 International Conference on Data Science, Agents & Artificial Intelligence (ICDSAAI), 2023, pp. 1–6

    2023

  16. [24]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, 2020, pp. 1877– 1901

    2020

  17. [25]

    Artificial intelligence and bias: Challenges, implica- tions, and remedies

    A. Min, “Artificial intelligence and bias: Challenges, implica- tions, and remedies.” Journal of Social Research, vol. 2, no. 11, 2023

    2023

  18. [26]

    Reformulating domain adaptation of large language models as adapt-retrieve-revise: A case study on Chinese legal domain,

    Z. Wan, Y . Zhang, Y . Wang, F. Cheng, and S. Kurohashi, “Reformulating domain adaptation of large language models as adapt-retrieve-revise: A case study on Chinese legal domain,” in Findings of the Association for Computational Linguistics: ACL 2024, L.-W. Ku, A. Martins, and...

    2024

  19. [27]

    Ai approaches to predictive justice: A critical assessment,

    F. Galli and G. Sartor, “Ai approaches to predictive justice: A critical assessment,” HUMANITIES AND RIGHTS GLOBAL NETWORK JOURNAL, vol. 5, no. 2, Nov. 2023. [Online]. Available: https://humanitiesandrights.com/journal/ index.php/har/article/view/118

    2023

  20. [28]

    Large language models are legal but they are not: Making the case for a powerful legalllm,

    T. Jayakumar, F. Farooqui, and L. Farooqui, “Large language models are legal but they are not: Making the case for a powerful legalllm,” arXiv preprint arXiv:2311.08890, 2023

    2023 arXiv

  21. [29]

    Uniform Crime Reporting (UCR) Program Data: Offenses Known and Clearances by Arrest, 2023,

    Federal Bureau of Investigation, “Uniform Crime Reporting (UCR) Program Data: Offenses Known and Clearances by Arrest, 2023,” https://ucr.fbi.gov/, 2024, accessed: 2024

    2023

  22. [2018]

    Available: https://journalistsresource.org/home/ study-highlights-need-for-public-health-approach-in-news \ -reporting-of-gun-violence/

    [Online]. Available: https://journalistsresource.org/home/ study-highlights-need-for-public-health-approach-in-news \ -reporting-of-gun-violence/

  23. [2023]

    Available: https://www.blocshop.io/blog/ chatgpt3-5-limitations?utm source=chatgpt.com

    [Online]. Available: https://www.blocshop.io/blog/ chatgpt3-5-limitations?utm source=chatgpt.com

  24. [2025]

    Available: https://aimlapi.com/comparisons/ chatgpt-4o-vs-o1-mini

    [Online]. Available: https://aimlapi.com/comparisons/ chatgpt-4o-vs-o1-mini

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.