{"id":"4daff126-e47b-4dd6-8f8e-39c85f59973a","arxiv_id":"2603.02150","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CrimeNER-db is a new, publicly released 1,568-document manually annotated corpus for crime-domain NER with a coarse/fine label hierarchy and zero-/few-shot benchmark results.","lead":"Researchers built and released CrimeNER-db, a manually annotated corpus of over 1,500 US crime and terrorism reports with coarse and fine-grained entity types. They benchmark zero- and few-shot NER models on it, reporting modest scores that they use to argue the dataset is usable for crime-text extraction.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No inter-annotator agreement is reported, so CrimeNERdb's 'high quality' claim rests on an unverified gold standard; a re-annotation reliability check would settle it.","rationale":"Reader's weakest assumption and mine coincide: no IAA. I considered dataset accessibility and taxonomy inconsistencies; both are important but secondary/fixable. The no-GitHub-link issue is easily remedied and does not affect the empirical quality claim. The 4/21 vs 5/22 vs 25 count inconsistencies are revealing but can be corrected in a revision; they do not by themselves invalidate the dataset. The core 'high quality' assertion is only as strong as the annotation standard, and the paper gives no quantitative evidence for it. A concrete re-annotation test would directly settle the concern. If it passes, the paper should be accepted; if it fails, the benchmark numbers cannot be trusted. The reader's CONDITIONAL verdict is appropriate; my analysis does not shift it, so I recommend UNCHANGED.","tokens_in":11931,"tokens_out":3215,"duration_ms":33244,"concrete_test":"Publish a reliability study on a random sample of at least 100 documents (or 500 sentences) from CrimeNERdb: have two or more independent annotators, blind to the original labels, annotate the same sample using the published guidelines; compute per-type and overall Cohen's kappa (or F1 agreement) for exact span boundaries and for span+label at both coarse and fine granularity; then adjudicate disagreements and compare the adjudicated labels to the released labels. If agreement is below about 0.8 kappa/F1 on fine-grained labels, the 'high quality' claim is unsupported and the reported benchmark numbers need re-evaluation with an agreed gold standard.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 1 contributions and Section 5) is that CrimeNERdb is a high-quality, first/biggest general-crime NER dataset. Section 3.2 describes annotation with Doccano and nested spans but reports no inter-annotator agreement, no adjudication, and no consistency check. The only evidence for annotation quality is the F1 scores of zero/few-shot models, yet those scores are computed against the same unvalidated labels; if two annotators would disagree substantially on span boundaries or on the fine-grained crime type (e.g., Burglar vs Other, or Criminal Person vs Criminal Organization), then every number in Tables 2–4 is uninterpretable as a measure of either model quality or dataset quality. The paper's own text admits the Crime type is 'more subject to interpretation' (§4.2), which makes the absence of reliability statistics particularly acute. Without evidence that the labels are reproducible, the 'high quality' conclusion is not supported, and the benchmark's validity as a resource for the community is in question. This is a fixable omission rather than a sign of fraud, but it is load-bearing: the resource's value depends on the gold standard being reliable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CrimeNERdb, a manually annotated corpus of 1,568 short documents (100–500 characters) drawn from U.S. Department of Justice press releases (2009–2018) and 2021 Global Terrorism Database reports. The annotation scheme has two levels — five coarse types (Crime, Actor, Agent, Agency, Logistic; the abstract counts vary) and a claimed 22 fine-grained types — and supports nested spans. The authors evaluate dataset quality via zero- and few-shot NER using NuNER, CONTaiNER, and several LLMs over 100 episodes (coarse and fine granularities), and conclude that the data are of high quality and constitute the first/biggest general-crime NER dataset, released on GitHub.","tokens_in":12263,"tokens_out":11792,"duration_ms":102103,"significance":"Should the resource hold up under scrutiny, it addresses a genuine gap: general (non-cyber) crime NER with realistic documents has no established public benchmark, and CrimeNERdb's 1,568 short documents plus a two-level, nested-capable taxonomy would be a useful testbed for zero/few-shot NER. The paper's strengths are that the evaluation uses well-known external baselines (NuNER, CONTaiNER, GPT-4o-Mini, GPT-4.1-Mini, Gemini-2.5-Flash) and public frameworks (LLMNER, Doccano), the 100-episode protocol with reported variance is a step toward reproducibility, and the database is claimed to be public. However, the central claim of annotation quality is load-bearing and currently rests on an unvalidated gold standard and a non-sequitur from model scores; the publication-worthiness of the benchmark depends on adding reliability evidence and fixing the internal inconsistencies.","major_comments":[{"comment":"The annotation-process section describes Doccano labeling of coarse/fine types and nested spans, but reports no inter-annotator agreement, no adjudication procedure, and no consistency statistics. Section 4.2 itself concedes that the Crime coarse type is 'more subject to interpretation and prone to small differences between the model predictions and the gold annotations.' Since every F1 in Tables 2–4 and the 'high quality' conclusion in Sections 1 and 5 are defined against this gold standard, the absence of reliability evidence makes the central claim unverifiable. Please add per-type (coarse and fine) span-level and label-level agreement measures (e.g., κ) from at least a double-annotation subsample, plus an adjudication protocol.","section":"§3.2"},{"comment":"The reasoning 'these models achieve a similar performance on Zero and Few-Shot settings compared to other benchmarks of Zero and Few-Shot NER, which shows that the annotated data is of great quality' is a non-sequitur. Reported F1 is 30–65 on coarse types and 10–35 on fine-grained types (Tables 2–4); nothing about these numbers or their parity with other benchmarks entails annotation quality, because model performance depends on task difficulty, model capacity, and label noise. No benchmark F1 values are actually quoted for comparison. Quality needs direct evidence (IAA/adjudication) or a controlled test (e.g., same models on perturbed/randomized labels). As written, the central quality claim is unsupported.","section":"§5 (and §1, contribution 3)"},{"comment":"CrimeNERdb explicitly contains nested spans ('District of Columbia' inside 'Attorney General for the District of Columbia'), but the paper never states how overlapping spans were scored in the zero/few-shot evaluations. Exact-match span F1 is undefined without a policy for nested/overlapping spans; the LLM and PLM baselines presumably emit flat spans. The reader cannot reproduce Tables 2–4 or interpret part-of-credit/penalty behavior. Please state the scoring protocol (e.g., best-match per template, outermost-only, token-level micro-F1) and whether the released data format preserves the nesting.","section":"§3.2 / §4"},{"comment":"Internal inconsistencies affect both taxonomy and results. (a) The fine-grained types listed in §3.2 sum to 25 (10 Crime, 4 Actor, 3 Agent, 3 Agency, 5 Logistic); the text instead reports 21 (arXiv abstract), 22 (intro, abstracts, §3.3, §5), and §5 says Crime has 8 fine types (which would total 23). (b) §4.2 states GPT-4o-Mini 'outperforms both CONTaiNER, GPT-4.1-Mini and Gemini-2.5-Flash on F1-score in 1 and 5-shot settings,' but Table 4 shows GPT-4.1-Mini with higher F1 (50.84 vs 42.55; 61.93 vs 60.09) and GPT-4o-Mini also not below Gemini in 5-shot only — and the 'Inter/Intra' columns are undefined. These contradictions must be resolved.","section":"§3.2–§3.3, §5; Table 4 vs §4.2"}],"minor_comments":[{"comment":"Pervasive typos and table-formatting defects: 'NER con general crime documents' (§3.3), 'enttity' and 'entiy spans' (§3.3), 'comparation' (§4.2); Table 2 renders 'Agent51.64'; Table 3 renders 'Crime 511.57±6.76' (missing space); Table 4's header shows six column labels with five data values and 'Inter/Intra' undefined; Fig. 3(b)'s caption attributes fine-grained zero-shot to GPT-4o-Mini while §4.1 states GPT-4.1-Mini was used for that experiment. A thorough proofreading pass is needed.","section":"§3.3, Tables 2–4, Fig. 3"},{"comment":"The contribution bullet reads 'manually annotated more than 1.5K sentences' while §3.3 and the abstracts say 1,568 'documents.' Since each selected item is a 100–500-character snippet, the document/sentence terminology should be aligned throughout.","section":"§1 (contributions)"},{"comment":"The 'first and biggest dataset' claim is asserted without a supporting comparison. Provide a table of existing crime/cybercrime NER corpora (e.g., APTNER, cyber-threat-intelligence datasets, FewNERD) with corpus sizes and entity counts, or hedge the claim to 'to our knowledge.'","section":"§1, §3.3"},{"comment":"NuNER is evaluated on a subset of entity types and on the whole dataset rather than per episode, so its F1 is not directly comparable with the LLM rows. The footnote discloses this, but the prose ('NUNER performs better in general') should explicitly warn the reader to avoid a comparative reading of Table 2.","section":"§4.1, Table 2"},{"comment":"The generation of the 100 episodes is underspecified: how were the support and query sets sampled (random vs class-balanced, query-set size)? The near-zero variance for GPT-4o-Mini (F1 σ=0.43 in Table 2) is surprising relative to other models and suggests near-identical episodes; please describe the episode construction and, if the variance is correct, explain it.","section":"§4.1"},{"comment":"The GitHub availability claim is not backed by a URL, license, data-format description, or reference to the annotation guidelines. For a dataset contribution, specify the release location, the export format (e.g., Doccano JSONL), and the license; also state explicitly how nested spans are represented in the released files.","section":"§3.2, §5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as an early draft: in addition to the load-bearing reliability gap, there are pervasive count inconsistencies and a direct prose/table contradiction. The dataset itself is plausibly useful and the issues are fixable, so I recommend major revision rather than rejection; the authors should also double-check Table 4's numbers and column definitions before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is CrimeNERdb, a new annotated corpus for general-crime NER with a two-level taxonomy and nested entities, drawn from DOJ press releases and the Global Terrorism Database. That is a genuine gap in the literature, and the resource looks useful for applied law-enforcement NLP. The annotation guidelines are described clearly, the examples in Table 5 are concrete, and the authors say the data is on GitHub. This is a solid resource paper, not a method paper — the zero-/few-shot evaluation is standard application of existing tools.\n\nThe soft spots are real but mostly addressable. The big one: no inter-annotator agreement is reported. The central claim that the database is 'high quality' is supported only by model F1 scores computed against the same unreliability-validated labels. Given the Crime type is described as 'more subject to interpretation,' IAA is load-bearing. Without it, every number in Tables 2–4 is uninterpretable as a measure of either model or dataset quality. The fix is straightforward: run a re-annotation reliability check on a subset and report Cohen's kappa or similar.\n\nThe count inconsistencies are sloppy and should be corrected before this goes out: the abstract says 4 coarse / 21 fine, the body says 5 / 22, and counting the listed fine types gives 25. That kind of mismatch undermines confidence even if it's just a copy-paste error.\n\nThe benchmark comparison is also informal — the conclusion that annotated data is 'of great quality' because models perform 'similarly to other benchmarks' is not a quantitative argument. A direct comparison to an established dataset using the same models would be more convincing. And omitting the Llama-2-7b results with a single sentence saying they were 'quite worse' is a minor reporting gap, but it is the kind of thing a skeptical reviewer will notice.\n\nWho is this for? Anyone building NER systems for crime, terrorism, or legal documents will want this dataset. Readers looking for method innovation will find little, but the resource itself is the point.\n\nMy take: this deserves a serious referee. The dataset is a plausible first-of-its-kind, and the issues are fixable with a revision. I would not trust the quality claim until IAA is reported, but I would use the dataset after that is settled.","headline":"A genuinely useful new crime-NER dataset, but the quality claim rests on an unverified gold standard; the missing inter-annotator agreement is fixable and essential.","tokens_in":12715,"tokens_out":1921,"would_cite":true,"duration_ms":20860,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CrimeNERdb is introduced as the first large benchmark for named-entity recognition on general crime documents, with 1,568 annotated texts.","keywords":["named-entity recognition","crime domain","dataset creation","zero-shot learning","few-shot learning","annotation","terrorism","law enforcement"],"falsifier":"A re-annotation of a random sample of CrimeNERdb documents by independent annotators, measuring inter-annotator agreement (e.g., F1 or Cohen's kappa on spans and labels). If agreement is low or near chance, the dataset's quality claim would be refuted.","tokens_in":11837,"feed_emoji":"🕵️","tokens_out":5217,"duration_ms":48394,"temperature":0.7,"pith_summary":"Law enforcement agencies need to extract entities like crimes, suspects, and logistics from text, but general crime-domain NER has lacked annotated data. This paper introduces CrimeNERdb, a publicly available dataset of 1,568 documents drawn from public press releases and terrorist event reports, annotated with a two-level hierarchy: 5 coarse entity types and 22 fine-grained types. The authors argue this is the first and largest such dataset. To demonstrate quality, they run zero-shot and few-shot NER experiments with state-of-the-art models, reporting scores comparable to other low-resource benchmarks. If the dataset is sound, it gives the NLP community a common testbed for crime-domain information extraction.","feed_headline":"Crime-domain NER gets its first 1,568-document dataset","feed_subtitle":"The public benchmark labels crimes, actors, and logistics in press releases and terror reports, enabling zero- and few-shot NER.","key_machinery":"The central object is the CrimeNERdb dataset itself, with its two-level entity annotation scheme (5 coarse types, 22 fine-grained types) and support for nested entities. The evaluation is built on an episodic few-shot protocol (n-way, k-shot) and zero-shot prompting, which serves as a surrogate for measuring annotation quality when no prior crime-domain NER benchmarks exist.","core_discovery":"CrimeNERdb contains 1,568 manually annotated documents with over 1,377 crime spans, 1,416 actor spans, and other entity types, including nested annotations. The authors claim it is the first and biggest dataset for NER on general crime documents and terrorist reports. They define a coarse hierarchy (Crime, Actor, Agent, Agency, Logistic) and fine-grained subtypes (e.g., Terrorism, Fraud, Criminal Person, Law Enforcement), and they validate the resource through experiments: fully supervised models, zero-shot LLM prompting, and few-shot episodic training. Their conclusion is that model performance on CrimeNERdb is comparable to other low-resource NER benchmarks, indicating the annotations are","pith_inferences":["Because the dataset draws only from English-language US sources, its entity taxonomy may be biased toward US legal and policing structures; a multilingual extension would need to adapt labels like 'Agency' and 'Agent'.","The reported F1 scores for Crime entities are lower than for Logistic, suggesting the crime-type definitions are the hardest to annotate consistently; future versions could benefit from more detailed guidelines or adjudication.","The absence of inter-annotator agreement statistics means the quality claim rests entirely on model performance; a small re-annotation study could quantify reproducibility."],"forward_implications":["CrimeNERdb provides a public benchmark for evaluating NER models on general crime text, filling a gap left by cybercrime-only datasets.","The two-level taxonomy (coarse and fine-grained) lets systems tune precision versus specificity, and supports nested NER evaluation.","Zero- and few-shot results on this dataset give a baseline for future low-resource crime-domain NER research.","The authors intend the dataset to support downstream tasks like crime-document classification and retrieval."],"fun_headline_variants":["CrimeNER: 1,568-doc public dataset for crime NER","CrimeNERdb: first general crime NER dataset","1,568 crime docs annotated for NER, zero-shot tested","CrimeNER: 21 fine-grained entity types from 1.5K docs","CrimeNERdb: 1.5K+ docs, 21 entity types, public"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The manual annotations are treated as a reliable gold standard, but the paper reports no inter-annotator agreement or adjudication; if annotators disagree substantially, the quality scores and the benchmark's validity become unclear.","fun_headline_variants_meta":{"raw":{"variants":["CrimeNER: 1,568-doc public dataset for crime NER","CrimeNERdb: first general crime NER dataset","1,568 crime docs annotated for NER, zero-shot tested","CrimeNER: 21 fine-grained entity types from 1.5K docs","CrimeNERdb: 1.5K+ docs, 21 entity types, public"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000221,"raw_usage":{"total_tokens":1257,"prompt_tokens":687,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":431,"completion_tokens_details":{"reasoning_tokens":468}},"tokens_in":431,"tokens_out":570,"duration_ms":4869,"temperature":1.0,"reasoning_tokens":468,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T19:24:55.394524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A re-annotation of a random sample of CrimeNERdb documents by independent annotators, measuring inter-annotator agreement (e.g., F1 or Cohen's kappa on spans and labels). If agreement is low or near chance, the dataset's quality claim would be refuted.","supporting_citations":[],"review_version":1}