Pith. sign in

REVIEW 3 major objections 4 minor 22 references

CrimeNER Demo: Named-Entity Recognition in the Crime Domain

T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A public platform now tags crime documents with 4 coarse and 21 fine entity types.

desk verdict Useful, honest demo paper for crime-domain NER; the platform is real but the F1 numbers should be treated as provisional pending external validation. read the letter →

arxiv 2607.14800 v1 pith:YZ3RNIRQ submitted 2026-07-16 cs.AI

Miguel Lopez-Duran , Julian Fierrez , Aythami Morales , Daniel DeAlcala , Gonzalo Mancera , Javier Irigoyen , Ruben Tolosana , Oscar Delgado
show 2 more authors
Francisco Jurado Alvaro Ortigosa
This is my paper · ORCID
classification cs.AI
keywords CrimeNERnamedentityrecognitionanalysisforensicdocumentfine-grainedtypestransformerlanguagemodelslawenforcementdemoplatform
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents CrimeNER Demo, a publicly available platform that automatically extracts crime-related entities from documents and labels them with two levels of granularity: 4 coarse types (Crime, Actor, Agent & Agency, Logistic) and 21 fine-grained subtypes (e.g., Terrorism, Criminal Organization, Weapons & Explosives). The central claim is that general-purpose transformer models, trained on the CrimeNER-db corpus of over 1.5K real-world documents, can annotate new documents at usable accuracy, with the best strict F1 around 0.65 and flexible F1 around 0.90 (XLM-RoBERTa-Base). The demo also lets users upload their own annotated data to fine-tune the models, so agencies working in other languages or crime types need only a small set of labeled examples. If true, this gives law enforcement and researchers a practical, adaptable tool for general crime NER—an area the paper identifies as underserved compared with legal-text and cyber-threat-specific NER.

What carries the argument

The carrying mechanism is the two-level entity hierarchy: each token is labelled as part of a coarse entity (Crime, Actor, Agent & Agency, or Logistic) and simultaneously as one of 21 fine-grained subtypes that contextualize it (e.g., Terrorism, Criminal Person, Law Enforcement, Location, Money). Extraction is done by transformer language models (XLM-RoBERTa, DeBERTa, RoBERTa, ALBERT, DistilBERT, BERT) pretrained on CrimeNER-db; the demo pipeline first extracts coarse spans, then derives fine entities from them, then injects colored highlights and JSON annotations into the document. The coarse-then-fine conditioning is what lets the system give two levels of detail from a single annotation p

What would settle it

Re-annotate a random 200-document sample of CrimeNER-db with independent annotators using the same 4/21 taxonomy; if per-entity-type inter-annotator agreement (e.g., Cohen's kappa) drops below ~0.6, or if XLM-RoBERTa-Base's strict F1 falls below 0.5 when evaluated on an independently authored corpus of crime reports with the same schema, the central claim that these models deliver general crime NER at ~0.65 strict F1 would be falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a two-level entity taxonomy—4 coarse and 21 fine-grained crime entity types—can be learned by standard transformer models from a moderately sized, real-world crime corpus, and that the resulting models transfer to a working demo platform. The paper reports that XLM-RoBERTa-Base achieves average strict F1 of 0.650 on both coarse and fine entities, with flexible (span-overlap) F1 around 0.90, based on a train/val/test split of CrimeNER-db. The system pipeline preprocesses documents, optionally fine-tunes on user-provided annotated data, extracts coarse entities first and then fine entities conditioned on them, and postprocesses the spans back into highlighted docu

Load-bearing premise

The weakest load-bearing assumption is that the CrimeNER-db ground-truth annotations are accurate and representative enough that the reported F1 scores (e.g., 0.650 strict) reflect real-world extraction quality—yet the paper gives no inter-annotator agreement and no independently curated test set.

Editorial extensions

If this is right

  • Law enforcement and researchers can run a public tool that turns raw crime documents into structured entity annotations without building a system from scratch.
  • Because users can upload their own annotated data, the same pretrained models can be adapted to other languages and crime types with fewer labeled examples than training from scratch, as the paper claims.
  • The reported strict-vs-flexible F1 gap (~0.65 vs ~0.90) means most errors are span-boundary misalignments rather than type confusions, suggesting a practical tolerance for highlighting use cases.
  • The pretrained models and demo repository give the research community a starting point and benchmark for general crime NER, complementing existing legal-text and cyber-threat-NER datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report inter-annotator agreement or external validation, so the numbers in Table 1 could be optimistic if CrimeNER-db annotations encode a particular reading of the taxonomy; a straightforward test would be to have independent annotators re-label a sample and measure agreement.
  • The flexible F1 of ~0.9 across all models hints that entity-type classification is near-saturated; the real bottleneck is exact span detection, so future work on boundary prediction may yield the biggest gains.
  • The claim that 'a small number of annotated samples is enough' is implicit and not yet benchmarked in this demo; measuring learning curves for fine-tuning with, say, 10/50/100 examples would make the claim concrete.
  • Cross-lingual and cross-jurisdiction transfer is untested; given that the corpus is US-focused, users in other legal systems may need to fine-tune, and the demo's success hinges on whether the taxonomy transfers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents CrimeNER Demo, a demonstrator platform for two-level named-entity recognition (NER) in the crime domain. It defines a 4-coarse / 21-fine entity taxonomy, describes the CrimeNER-db dataset of about 1.5K annotated documents, and offers six pretrained transformer-based NER models plus a user-facing fine-tuning option. The platform takes input documents, applies a coarse and fine entity extraction pipeline, and returns highlighted annotations with optional JSON output. The only quantitative evaluation is Table 1, reporting strict and flexible F1 scores on the authors' own test split; the best strict coarse F1 is 0.650 (XLM-RoBERTa-Base). The demo, tutorial, and video are publicly available on GitHub.

Significance. If the reported results hold, this is a useful public resource: it targets a relatively underserved NER domain, provides a two-level annotation granularity, and includes a practical fine-tuning path for downstream users. The paper ships concrete artifacts (code, tutorial, video, multiple pretrained models), which is a real strength. However, the claimed extraction quality rests entirely on Table 1, and that evidence is weakened by the absence of inter-annotator agreement, external baselines, variance estimates, and a known PDF-alignment defect that directly affects the user-visible output. The fine-tuning benefit is also asserted rather than demonstrated. The platform itself is credible as a demonstration; the quantitative and deployed-quality claims need strengthening.

major comments (3)
  1. [Section 2, Table 1] Table 1 is the sole quantitative support for the paper's central extraction-quality claim. It reports no error bars, number of runs, or random-seed variance, and CrimeNER-db is annotated by the authors under their own novel taxonomy. Without inter-annotator agreement statistics, annotation guidelines, or an external validation corpus, the strict F1 values around 0.65 may reflect annotation/schema bias rather than real-world extraction quality. Please provide (i) annotation guidelines and IAA on a sample; (ii) standard deviations across seeds; (iii) a per-entity-type breakdown; and (iv) ideally a comparison on an external crime/legal NER benchmark or a qualitative error analysis.
  2. [Section 4, demonstrator description] The paper acknowledges: 'There is some misalignment between the annotation and the PDF text, as the positions in the PDF metadata are not exactly aligned with the visual text position in the document.' This is a known failure mode in the primary user-facing output: even a correct model prediction can be rendered at the wrong visual location, so the demonstrated annotations may mislead users. Since the platform's promise is that users 'receive the input documents with the extracted crime entities annotated and highlighted,' this caveat needs to be quantified (e.g., percentage of documents/spans affected) or addressed, rather than mentioned in passing.
  3. [Section 1 and Section 3.1 (Specific Fine-tuning)] The introduction claims that 'only a small number of annotated samples is enough to extract meaningful crime information,' and the system offers a fine-tuning module, but no experiment, ablation, or case study supports this claim. As presented, the fine-tuning path is a UI feature rather than an evaluated capability. Either soften the claim or report a small pilot study (e.g., fine-tuning on a few dozen sentences from a target corpus) that demonstrates the expected benefit.
minor comments (4)
  1. [Figure 1 and Section 2] The text merges 'Agent' and 'Agency' into a single coarse type 'Agent & Agency' (footnote 2), but Figure 1's legend lists 'Agent' and 'Agency' as separate colors. Align the figure with the taxonomy.
  2. [Section 2 and Section 4] CrimeNER-db is described in Section 2 as 'terrorist reports or real press notes from the US' but in Section 4 as 'real-world documents from the U.S. Department of Justice and other terrorist and crime reports.' Clarify the actual sources.
  3. [Table 1] The identical reported values for XLM-RoBERTa-Base coarse strict F1 (0.650) and fine strict F1 (0.650) look coincidental; consider reporting more significant digits or noting the coincidence.
  4. [General] There are minor typographical issues (e.g., 'arenotsuitable' in the introduction, missing space). A proofreading pass is recommended.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; evaluation is supervised against human annotations; concerns are validity, not circularity.

full rationale

The paper presents a demo platform and reports F1 scores from training and testing NER models on CrimeNER-db. There is no mathematical derivation that reduces to its own inputs. The F1 scores are empirical measurements on a held-out test split, not fitted parameters renamed as predictions. The CrimeNER-db dataset is cited from the authors' prior work, but the paper does not derive the dataset's correctness from the model's performance; the annotations are treated as ground truth. The absence of inter-annotator agreement and external validation is a legitimate concern about external validity, but it is not circularity under the enumerated patterns. The §4 caveat about PDF-text misalignment is a practical limitation, not a circular step. The self-citation is load-bearing in the sense that the benchmark comes from the same group, but it is not an unverified uniqueness theorem or ansatz, and the evaluation is still a supervised measurement. Thus score 1 reflects a minor concern about the closed evaluation loop, not significant circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central performance claims rest on the quality of the authors' own CrimeNER-db annotations and the adequacy of their entity taxonomy; neither is externally validated. Model hyperparameters are unreported, and the few-shot fine-tuning benefit is an unverified assumption.

free parameters (1)
  • Model training hyperparameters (learning rate, batch size, epochs, seeds) = not reported
    The six models in Table 1 were trained on CrimeNER-db, but no hyperparameters are given; F1 scores depend on these choices, so results are not exactly reproducible from the paper.
assumptions (3)
  • domain assumption CrimeNER-db annotations are accurate ground truth
    All F1 scores in Table 1 are computed against this annotation; no inter-annotator agreement or external quality check is reported (Section 2, Table 1).
  • ad hoc to paper The 4-coarse/21-fine entity taxonomy covers general crime information
    The taxonomy is defined by the authors in the companion dataset paper [13] and asserted to be informative; no user study or coverage analysis is provided (Section 2).
  • ad hoc to paper A small number of user-annotated samples suffices for useful fine-tuning
    Stated in Section 1 ('only a small number of annotated samples is enough') without experimental evidence; this underpins the platform's main customization feature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CrimeNER Demo: Named-Entity Recognition in the Crime Domain." pith.science (2026). https://pith.science/paper/YZ3RNIRQ

@misc{pith2026260714800,
  author       = {Pith},
  title        = {Pith review of: CrimeNER Demo: Named-Entity Recognition in the Crime Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZ3RNIRQ}},
  note         = {Machine review of arXiv:2607.14800}
}
read the original abstract

We present CrimeNER Demo, an AI-powered platform that enables us to extract general crime-related information from documents and classify them into entity types with two levels of granularity. We provide pretrained NER models on the CrimeNER database, and we give the possibility to users to provide their own annotated data to train models for their own specific cases. This demonstrator aims to promote crime-related NER research and provides a practical tool to automatically extract crime information for researchers and law enforcement agencies. The demonstrator includes: i) Pretrained NER models on the crime domain; ii) Possibility to finetune the models on specific data annotated by the user; and iii) An automatic pipeline to extract and annotate crime entities from documents. The demo platform, a tutorial to run the demo, and a video demonstration are publicly available on GitHub.

Figures

Figures reproduced from arXiv: 2607.14800 by the authors.

Figure 1
Figure 1. Example of different annotations for a sample document using the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of the CrimeNER Demo. Documents are processed with the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 5 linked inside Pith

  1. [1]

    In: Proc

    Au, T.W.T., Lampos, V., Cox, I.: E-NER: An annotated named entity recognition corpus of legal text. In: Proc. of the NLP Workshop. pp. 246–255 (2022)

    2022

  2. [2]

    In: ACL (2020)

    Conneau, A., et al.: Unsupervised cross-lingual representation. In: ACL (2020)

    2020

  3. [3]

    In: arXiv preprint (2026)

    DeAlcala, D., Ko, C.Y., Morales, A., Fierrez, J., et al.: DocAI: A framework for generating and identifying AI-edited documents. In: arXiv preprint (2026)

    2026

  4. [4]

    In: IEEE COMPSAC (2026)

    DeAlcala, D., Mancera, G., Fierrez, J., et al.: Is my vision-language data in your AI? Membership Inference Test (MINT) Demo 2. In: IEEE COMPSAC (2026)

    2026

  5. [5]

    CoRRabs/1810.04805(2018)

    Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirec- tional transformers for language understanding. CoRRabs/1810.04805(2018)

    arXiv 2018

  6. [6]

    In: Proc

    Ding, N., Xu, G., Chen, Y., Wang, X., Han, X., et al.: Few-NERD: A few-shot named entity recognition dataset. In: Proc. ACL. pp. 3198–3213 (2021)

    2021

  7. [7]

    In: ICLR (2021)

    He, P., et al.: DeBERTa: BERT with disentangled attention. In: ICLR (2021)

    2021

  8. [8]

    In: IEEE ICCST (2026)

    Irigoyen, J., et al.: Overview of risk assessment and management for intelligent systems under the AI Act and beyond. In: IEEE ICCST (2026)

    2026

Show all 22 references
  1. [9]

    In: IEEE Intl

    Korshunov, P., et al.: DeepID challenge of detecting synthetic manipulations in ID documents. In: IEEE Intl. Conf. on Computer Vision Workshops (2025)

    2025

  2. [10]

    CoRRabs/1909.11942(2019)

    Lan, Z., Chen, M., Goodman, S., Gimpel, K., et al.: ALBERT: A lite BERT for self- supervised learning of language representations. CoRRabs/1909.11942(2019)

    1909 arXiv

  3. [11]

    CoRRabs/1907.11692(2019)

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., et al.: RoBERTa: A robustly optimized BERT pretraining approach. CoRRabs/1907.11692(2019)

    1907 arXiv

  4. [12]

    In: ICDAR Workshops (2025)

    Lopez-Duran, M., Fierrez, J., et al.: Benchmarking graph neural networks for doc- ument layout analysis in public affairs. In: ICDAR Workshops (2025)

    2025

  5. [13]

    arXiv:2603.02150 (2026)

    Lopez-Duran, M., Fierrez, J., et al.: Named-entity recognition in the crime domain (CrimeNER): Case study and dataset. arXiv:2603.02150 (2026)

    2026 arXiv

  6. [14]

    In: ICDAR Workshops (2026)

    Lopez-Duran, M., Marrero, E., et al.: Comparative study of domain-adapted VLMs for general document visual question answering. In: ICDAR Workshops (2026)

    2026

  7. [15]

    In: IEEE COMPSAC (2026)

    Mancera, G., DeAlcala, D., Morales, A., Fierrez, J., et al.: Auditing training data in domain-adapted LLMs: LoRA-MINT. In: IEEE COMPSAC (2026)

    2026

  8. [16]

    In: ICDAR Workshops (2025)

    Mancera, G., Morales, A., Fierrez, J., et al.: PBa-LLM: Privacy-and bias-aware NLP using Named-Entity Recognition (NER). In: ICDAR Workshops (2025)

    2025

  9. [17]

    Muñoz, J., et al.: Privacy-aware detection of fake identity documents: methodology, benchmark, and improved algorithms (FakeIDet2). Inf. Fusion128, 103969 (2026)

    2026

  10. [18]

    SN Computer Science4(5), 434 (June 2023)

    Peña, A., et al.: Human-centric multimodal machine learning: Recent advances and testbed on AI-based recruitment. SN Computer Science4(5), 434 (June 2023)

    2023

  11. [19]

    Peña, A., et al.: Continuous document layout analysis: Human-in-the-loop AI cura- tion, database & evaluation in the domain of public affairs. Inf. Fusion108(2024)

    2024

  12. [20]

    ArXivabs/1910.01108(2019)

    Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXivabs/1910.01108(2019)

    1910 arXiv

  13. [21]

    Song, B., et al.: Deep learning methods for biomedical named entity recognition: a survey and qualitative comparison. Brief. Bioinform.22(6), bbab282 (2021)

    2021

  14. [22]

    In: IEEE CSCWD

    Wang, X., He, S., Xiong, Z., Wei, X., et al.: APTNER: A specific dataset for NER missions in cyber threat intelligence field. In: IEEE CSCWD. pp. 1233–1238 (2022)

    2022

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.