{"id":"83ebae29-ae99-4722-aa5e-9bdcfa899199","arxiv_id":"2504.12545","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A comparison of GPT-3.5, GPT-4o, and o1-mini on few-shot named entity recognition over a new 153-event mass-shooting dataset finds GPT-4o most accurate, though the dataset and evaluation details are not fully disclosed.","lead":"This paper introduces a new annotated dataset of 153 mass-shooting event summaries and tests three OpenAI language models (GPT-3.5, GPT-4o, o1-mini) on extracting legal and investigative entities. GPT-4o gets the top micro F1 score, but the dataset is not released and the claim that more examples always improve every model is contradicted by the paper's own table.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GPT-4o ranking in Table III depends on an undocumented evaluation split and output-parsing rule; if shots are drawn from the same 153 summaries being scored, the reported Micro-F1 (0.6823) may reflect prompt/test overlap rather than generalization.","rationale":"I read the paper as an empirical comparison of three LLMs on a new NER dataset, with the central claim that GPT-4o is the most effective model for mass-shooting NER. For that claim to hold, the evaluation must separate in-context exemplars from test instances and score model outputs against gold spans using a consistent rule. The paper never states that such a separation exists. This is a correctness risk, not a style issue: if the same 153 summaries were used to build the few-shot prompts and to compute Table III, the scores would be inflated in a way that could change the model ranking. The reader's weakest assumption was annotation reliability; I agree that is important, but the more immediate logical gap is the absence of any documented split and output-parsing pipeline. These are addressable: the authors can release their split, prompts, and scoring code, or rerun with a held-out event set. My verdict remains CONDITIONAL because the concern is concrete and fixable; if the proposed check reveals leakage, the appropriate verdict would be REJECT. I therefore partially agree with the reader's assessment: we identify the same family of evaluation-validity concerns, but the specific weakest link is the missing held-out evaluation rather than inter-annotator agreement alone.","tokens_in":9483,"tokens_out":4614,"duration_ms":53251,"concrete_test":"Request or reconstruct the exact prompt set and verify whether any evaluated Mother Jones summary appears as a few-shot exemplar. Then rerun the 14-shot comparison under a strict event-level split: randomly hold out 30 of the 153 summaries for testing, use only the remaining 123 for constructing shots, and score with an exact span/type match after applying the stated parsing rule. If held-out Micro-F1 for GPT-4o drops materially below 0.6823, or if another model surpasses GPT-4o on this split, then Table III's ranking is an artifact of prompt/test overlap rather than a valid comparison of NER capability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—GPT-4o is the most effective model for mass-shooting NER, with the highest Micro-F1 of 0.6823 at 14 shots (Table III)—is only as strong as the evaluation protocol, and that protocol is nowhere specified. Section III describes a corpus of 153 Mother Jones summaries, and Section IV-B says each shot 'illustrates an input snippet paired with the desired output' drawn from the annotation schema, but no train/test split, cross-validation, or held-out set is described anywhere between Section III and Table III. If the few-shot exemplars are selected from the same 153 event summaries that are then scored, every reported number in Table III and Figure 4 is contaminated by the model having seen near-duplicate text during in-context learning. The paper also omits the output-parsing step: raw generative LLM responses must be converted into entity spans/labels before micro precision/recall can be computed, and no exact/partial matching rule is given. A second-order issue is gold-label reliability: no inter-annotator agreement is reported, and the label inventory in the prompt (which includes OPST, VE, Money, Percent, Art, Language) does not match Tables I–II, so it is unclear what the models were actually scored against. All three issues are fixable, but until the split and scoring protocol are provided, the ranking is not independently checkable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a mass-shooting event NER dataset built from 153 Mother Jones event summaries, with a four-category, 41-subtype annotation schema (Offense, Offender, Victim, Environment/Justice Agency), and evaluates GPT-3.5, GPT-4o, and o1-mini under 5-, 10-, and 14-shot prompting. The reported results in Table III show GPT-4o achieving the highest Micro Precision, Micro Recall, and Micro F1 across prompt sizes, with o1-mini as a competitive resource-efficient alternative. The authors claim this is the first dataset for knowledge acquisition on mass-shooting events using NER and that increasing shot counts improves all models, with larger gains for GPT-4o and o1-mini.","tokens_in":9806,"tokens_out":3366,"duration_ms":36117,"significance":"If the evaluation protocol is made fully transparent and the gold-standard annotations are validated, the released dataset would fill a genuine gap: there is little prior work on structured knowledge acquisition for mass-shooting events, a domain with clear public-safety and legal-investigation relevance. The paper also offers a useful head-to-head comparison of three commercial LLMs on a low-resource, domain-specific NER task, which is of practical value. The schema is thoughtfully organized and the use of a real-world corpus is a strength. However, the current significance is substantially limited by three load-bearing omissions: the train/test or exemplar/evaluation split is not described, the output-parsing and span-matching rule is not specified, and the reliability of the self-annotated gold labels is not quantified. As written, the central ranking in Table III is not independently checkable, and one of the paper's headline claims is contradicted by its own numbers for GPT-3.5.","major_comments":[{"comment":"The paper claims that \"increasing the shot count enhances the performance of all models,\" but this is contradicted by the GPT-3.5 row: Micro F1 falls from 0.4436 at 10 shots to 0.4245 at 14 shots, and Micro Recall falls from 0.3445 to 0.3241. The text later acknowledges only a \"marginal improvement\" in recall, which itself is not supported by these numbers. This claim appears in the abstract, Section I, Section V-B, and Section VII, so it needs to be corrected or carefully qualified.","section":"Section V-B, Table III"},{"comment":"The evaluation protocol does not specify how the few-shot exemplars are separated from the instances being scored. Section III-A describes a corpus of 153 event summaries, and Section IV-B says each shot illustrates an input snippet paired with the desired output, but no train/test split, cross-validation, or held-out set is described. If the exemplars are drawn from the same 153 summaries that are later scored, the Table III numbers may reflect prompt/test overlap rather than generalization. The authors should specify the split, the exemplar-selection procedure, the number of independent runs, seeds, and whether results are averaged over prompt orderings.","section":"Section IV-B and Section V-B"},{"comment":"No output-parsing or span-matching rule is given. The models generate free-form text, but micro precision and micro recall for NER require converting these responses into entity spans and labels and comparing them with gold spans using a specified matching criterion (exact match, partial match, label-only, or other). Without this information, the numbers in Table III cannot be reproduced or independently verified. The authors should describe the parsing step and the exact matching rule used for scoring.","section":"Section V-A and Section V-B"},{"comment":"The gold-label reliability is not demonstrated. Section III-B reports that two student annotators worked under expert supervision but gives no inter-annotator agreement measure (e.g., Cohen's kappa or F1 agreement), and the annotation guidelines are only summarized in Tables I and II. In addition, the prompt schema in Figure 3 lists labels such as OPST, VE, Money, Percent, Art, and Language that do not appear in Tables I-II, so it is unclear what label inventory the models were actually scored against. The authors should report annotator agreement and reconcile the label set between the prompt and the gold-standard schema.","section":"Section III-B and Figure 3 prompt schema"}],"minor_comments":[{"comment":"Several quantitative claims in Related Work are stated without specific supporting citations, including \"increase recall by approximately 5%,\" \"F1-scores exceeding 93%,\" and a \"10% reduction in entity disambiguation errors.\" These need explicit references or should be removed.","section":"Section VI-B"},{"comment":"The per-entity F1 comparison in Figure 4 is difficult to read for low-frequency labels, and the text makes claims such as \"up to 4-times higher F1-scores\" without reporting the underlying values. A supplementary table with numerical F1 per entity type would make the claims checkable.","section":"Figure 4"},{"comment":"Section III-B says annotations follow the standard BIO format, but the prompt in Figure 3 asks for generative outputs listing key entities. The relationship between the BIO-annotated gold data and the generative few-shot format should be clarified.","section":"Section III-B and Section IV-B"},{"comment":"Some references appear mismatched to the claims they support; for example, reference [24] is cited for the claim that NLP models achieve over 90% accuracy in extracting case elements from court transcripts, but [24] is the GPT-3 paper and does not appear to contain that result. Please verify all forward references.","section":"References"},{"comment":"The manuscript has minor typographical and style inconsistencies, such as inconsistent hyphenation of \"mass shooting\" versus \"mass-shooting,\" inconsistent capitalization of \"Micro recall\" versus \"Micro Recall,\" and occasional tense inconsistencies in the abstract and introduction.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core contribution is potentially useful, but the manuscript as posted lacks the experimental details needed to verify its central claim. The novelty claim of being the \"first dataset\" for mass-shooting NER should be checked against existing legal and forensic NER resources during revision. I also note that several model descriptions cite blog posts and Wikipedia; for a journal version, the authors should prefer primary or peer-reviewed sources. The paper is within scope for a computational social science / AI-for-justice venue, provided the evaluation protocol is completed and the claims are adjusted to match the reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a niche but legitimate dataset paper with a messy evaluation section. The main contribution is a 41-type NER annotation schema for mass-shooting event summaries, applied to 153 Mother Jones accounts. That, plus the three-round annotation procedure, is the real asset. The GPT-4o vs o1-mini vs GPT-3.5 comparison is a reasonable first look, and I don't doubt the ranking: GPT-4o ahead on all three micro metrics at every shot count, o1-mini in the middle, GPT-3.5 clearly behind.\n\nThe soft spots are real. First, the abstract and Section V-B claim that increasing shots helps all models, but Table III shows GPT-3.5's F1 going 0.4043 -> 0.4436 -> 0.4245 as shots go 5 -> 10 -> 14, and recall 0.3062 -> 0.3445 -> 0.3241. That's not 'enhances all models.' The text even acknowledges GPT-3.5's recall only 'rises slightly' from 5 to 14, but the 14-shot drop vs 10-shot is ignored. This needs a rewrite and honestly weakens trust in the analysis.\n\nSecond, the evaluation protocol is under-specified. No train/test split is described anywhere; the few-shot exemplars may be drawn from the same 153 summaries being scored. If so, the models have seen near-duplicate text in context, which inflates every number. The prompt in Figure 3 also lists entity types not in Tables I–II (OPST, VE, Money, Percent, Art, Language), so it is unclear what label space the models were actually scored against. No parsing rule is given for converting generative output to spans, and there is no inter-annotator agreement. All of this is fixable, but as written the ranking is not independently checkable.\n\nThere are also citation mismatches in Related Work (the names in the text don't match the references), no classic NER baseline, no error bars, and no released data. These are less serious but should be cleaned up.\n\nVerdict: the dataset is the contribution. The LLM comparison is secondary and currently not reliable as evidence for the 'best model' claim. If the authors clarify the split, parsing, and label space, and fix the shot-count contradiction, this is a solid workshop/short-paper contribution for AI-for-justice or computational criminology audiences. I would send it to peer review, but flag that the empirical claims need major revision.","headline":"A niche NER dataset for mass-shooting events with a useful annotation schema, but the LLM comparison is under-specified and one headline claim contradicts its own table.","tokens_in":10318,"tokens_out":3446,"would_cite":false,"duration_ms":31850,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4o, prompted with a few labeled examples, is the most effective model for extracting offender, victim, location, and weapon entities from mass-shooting event text, and the paper introduces the first dataset for this task.","keywords":["named entity recognition","mass-shooting events","large language models","few-shot prompting","AI-driven justice","knowledge acquisition","GPT-4o","legal information extraction"],"falsifier":"Have a second, independent team re-annotate the same 153 event summaries using the same 41-label guidelines, then re-run the three models with the same prompts; if the relative ranking changes or the agreement with the original labels is low, the reported GPT-4o advantage would not survive.","tokens_in":9324,"feed_emoji":"⚖️","tokens_out":6216,"duration_ms":54331,"temperature":0.7,"pith_summary":"The paper sets out to automate knowledge acquisition from mass-shooting reports using named entity recognition, so that investigators and policymakers can quickly get structured facts such as who the offender was, how many victims died, and what weapon was used. Its central claim is that a 41-label annotation schema covering offender, victim, environment, and justice-agency information can be applied to 153 real-world event summaries, and that GPT-4o with few-shot prompting extracts these entities more reliably than GPT-3.5 or o1-mini. If true, this gives AI-driven justice a reusable benchmark and a practical recipe for pulling structured leads from unstructured crisis text.","feed_headline":"GPT-4o beats rivals at extracting facts from mass-shooting reports","feed_subtitle":"Using a handful of examples, the model finds offenders, victims, weapons, and locations in real events.","key_machinery":"The carrying mechanism is few-shot prompting for generative NER, adapted from the GPT-NER approach: the task is reframed from sequence labeling into a text-generation problem where the prompt lists 41 entity types with short labels (ON, OA, CI, TPE, etc.) and supplies a handful of example input–output pairs. The LLM then produces structured entity annotations directly, without fine-tuning, and the annotations are scored with standard Micro Precision, Recall, and F1 metrics. This machinery lets the authors test how much in-context learning alone can transfer a general model to a specialized legal-investigative domain.","core_discovery":"On the paper's own terms, the discovery is that large language models can perform domain-specific NER in a low-resource setting when given a carefully designed prompt that lists the target entity types and a few annotated examples. The authors report that GPT-4o achieves the highest Micro F1-score (0.6823 with 14 shots), with o1-mini as a close, cheaper alternative, and that increasing the number of shots improves all models, with the largest gains for GPT-4o and o1-mini. They also contribute what they call the first mass-shooting NER dataset, built from public event summaries and annotated under expert supervision with 41 entity subcategories.","pith_inferences":["A natural next check is whether the same prompt-and-shot recipe transfers to other crisis domains, such as active-shooter 911 transcripts or court filings; the paper's schema would probably need new labels like 'legal charge' or 'evidence type'.","Because the paper does not report inter-annotator agreement, the true ceiling for these models may be lower or higher than the reported F1; an independent reannotation of the same 153 summaries would settle whether the leaderboard is meaningful.","The shot-count trend suggests that a small amount of curated, expert-annotated examples could unlock high-quality extraction in other low-resource legal domains, so the main bottleneck is annotation cost, not model capacity."],"forward_implications":["GPT-4o can be used as a drop-in extractor for offender names, victim counts, criminal instruments, and other entities from mass-shooting narratives, reaching about 0.68 Micro F1 with 14 examples.","The 41-label schema and the 153-event corpus provide a reusable starting point for building knowledge graphs of mass-shooting events.","o1-mini offers a resource-efficient option for simpler NER tasks, with precision close to GPT-4o but lower recall.","Adding more in-context examples helps all evaluated models, but the benefit is notably larger for GPT-4o and o1-mini, suggesting these models adapt better to few-shot learning."],"supporting_citations":[{"why":"Supplies the GPT-NER method that converts sequence labeling into a generative task with special token cues, forming the basis for the paper's prompt schema.","marker":"[5]"},{"why":"Provides the survey of prompting techniques that justifies few-shot prompting for in-context learning.","marker":"[10]"},{"why":"Shows that pre-trained language models can be made better few-shot learners, supporting the claim that a handful of labeled examples suffices.","marker":"[19]"},{"why":"Documents sensitivity of LLMs to prompt design, motivating the paper's careful prompt engineering.","marker":"[20]"},{"why":"Supplies the standard NER evaluation metrics (F1, Micro Precision, Micro Recall) used to score the models.","marker":"[21]"}],"fun_headline_variants":["GPT-4o wins at pulling key entities from shooting reports","Few-shot LLMs find offenders, weapons in mass-shooting data","New dataset trains LLMs to extract shooting case entities","GPT-4o and o1-mini lead few-shot shooting NER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the annotations produced by two student labelers under expert supervision are correct and complete enough to serve as ground truth, but it reports no agreement measure between the labelers.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o wins at pulling key entities from shooting reports","Few-shot LLMs find offenders, weapons in mass-shooting data","New dataset trains LLMs to extract shooting case entities","GPT-4o and o1-mini lead few-shot shooting NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0011,"raw_usage":{"total_tokens":4575,"prompt_tokens":914,"completion_tokens":3661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":3588}},"tokens_in":530,"tokens_out":3661,"duration_ms":21720,"temperature":1.0,"reasoning_tokens":3588,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:28:02.659553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a second, independent team re-annotate the same 153 event summaries using the same 41-label guidelines, then re-run the three models with the same prompts; if the relative ranking changes or the agreement with the original labels is low, the reported GPT-4o advantage would not survive.","supporting_citations":[{"cited_title":"A survey on deep learning for named entity recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the standard NER evaluation metrics (F1, Micro Precision, Micro Recall) used to score the models."}],"review_version":1}