Pith. sign in

REVIEW 4 major objections 5 minor 56 references

Survey: Understand the challenges of MachineLearning Experts using Named EntityRecognition Tools

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A survey of ML experts finds performance is the top criterion for choosing NER tools.

desk verdict Small, clearly reported expert survey on NER tool choice; new primary data but the sample undercuts the 'ML expert' label. read the letter →

arxiv 2501.16112 v1 pith:XB4ZQOWM submitted 2025-01-27 cs.IR cs.CL

Florian Freund , Philippe Tamla , Matthias Hemmje This is my paper · ORCID
classification cs.IRcs.CL
keywords ExpertSurveyNamedEntityRecognitionMachineLearningToolSelectionEvaluationCriteriaCloudComputingNaturalLanguageProcessing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a survey of 23 people who work with machine learning, asking how they compare and choose named-entity-recognition (NER) tools. Its aim is to establish which selection criteria matter most and which obstacles are the most hindering, so that future software can help non-experts make the same choices. The results say performance is the dominant criterion, but every listed criterion was rated important or very important by at least one respondent, meaning the right tool depends on the project. The paper also distinguishes cloud-based services from locally installed tools: cost and user-friendliness drive cloud choices, while documentation and the effort of learning a new system matter most for local tools. A sympathetic reader would take the study's contribution to be a criterion list plus deployment-specific priorities, not a general ranking of tools.

What carries the argument

The load-bearing mechanism is the per-tool questionnaire: for each NER tool a respondent had used, the survey asked them to rate nine selection criteria and six potential hindrances on five-point scales, then aggregated the ratings across respondents. This design lets the authors split the same criteria by cloud-based versus locally installable tools and compute average-priority differences, which is what produces the deployment-specific conclusions. The per-tool structure was itself a product of pilot testing, after an earlier global question proved unable to link a criterion like performance or privacy to a specific tool choice.

What would settle it

A replication survey using the same per-tool criteria questions with a larger sample of ML experts who each have more than three years of NER experience would settle the priority claim if performance no longer ranked first overall, or if cloud users did not rate cost and ease of use above local-tool concerns.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that ML experts evaluate NER tools primarily by performance, with an average importance rating of 4.57 on a 5-point scale, while no criterion can be dismissed as universally unimportant. When the responses are split by deployment type, cloud-based services are judged especially on user interface and ease of use (average 4.43, a 1.09-point gap over local tools) and on licensing and cost, whereas locally installed systems are judged especially on documentation and support (a 1.14-point gap favoring local tools) and on minimizing the time and effort needed to learn the framework. The survey also finds that the most frequently hindering challenge is the time and effort to learn a new framework, and that locally operated open-source large language models are already used for NER and should be included in future tool comparisons.

Load-bearing premise

The load-bearing premise is that 23 volunteers who answered an emailed survey, half of whom reported less than one year of NER experience, can speak for the challenges of experienced ML experts; the paper's own sampling plan says no representative cross-section was required.

Editorial extensions

If this is right

  • Any support system for choosing NER tools must let users weigh multiple criteria, because all nine surveyed criteria were rated important or very important at least once; performance alone does not decide.
  • For cloud-based NER offerings, controlling cost and simplifying the user interface are concrete levers that would address the two highest-priority cloud concerns.
  • For locally installed NER tools, improving documentation and reducing the effort of learning the framework would remove the most hindering obstacle.
  • Locally operated open-source large language models should be treated as a legitimate NER option in comparisons and selection aids.
  • Future surveys should recruit domain experts outside computer science, since the paper itself cautions that its results cannot be directly generalized to them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the project-specific spread of criteria suggests a configurable comparison matrix rather than a single recommended tool; a decision aid would need to elicit the user's deployment context first.
  • Editorial inference: the high priority given to performance may partly reflect the respondents' computer-science background; a domain-expert sample could shift priorities toward privacy and knowledge-domain fit, and the paper's own data leave this open.
  • Editorial inference: the finding that open-source local LLMs are used for NER, combined with the cost barrier for cloud services, points toward a two-tier selection space where local open models and paid cloud APIs serve different users; this distinction could be tested by asking respondents directly which tool class they ended up choosing.
  • Editorial inference: a testable extension would ask respondents to rank tools for a fixed hypothetical task, which would disentangle project-specificity from personal preference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a survey of 23 self-selected participants, recruited by email and through a research institute mailing list, about how they compare and select Named Entity Recognition (NER) tools and frameworks. The study is organized around two research questions: which criteria ML experts use to evaluate NER tools (RQ1) and what challenges they face when selecting a tool (RQ2). Using Kasunic's survey methodology, the authors designed a questionnaire, piloted it with three experts, and analyzed responses with descriptive statistics and figures. The main reported findings are that performance is the most important selection criterion, that all listed criteria are considered important by at least some respondents, that cloud-based services are particularly affected by cost and user-friendliness, and that locally installed tools are particularly affected by the time and effort needed to learn the system. The paper concludes with design implications for a future system to support non-experts in choosing NER tools.

Significance. If the findings held up, this would fill a genuine gap: most prior NER tool comparisons are benchmark studies, and the authors are right that user-centered evidence on how practitioners choose NER tools is scarce. The paper has real strengths: it follows a recognized survey methodology, reports a pilot test, makes the questionnaire publicly available, provides primary data, and explicitly discusses several threats to validity. The authors are also appropriately cautious in places, noting that the computer-science-heavy sample limits generalization to domain experts. However, the contribution is currently tentative because the respondents do not consistently match the paper's own definition of an ML/NER expert, the sample is small (N=23), several per-tool and per-category analyses rest on one to nine responses, and no inferential statistics or uncertainty measures are provided. The descriptive patterns are plausible and useful as hypothesis-generating evidence, but they are not yet sufficient to support the paper's stronger conclusions about what 'ML experts' need.

major comments (4)
  1. [Section 3 (Target Audience) and Section 5.1 (Figures 5 and 6)] The paper's stated inclusion criterion for experts is more than three years of experience in ML, NLP, and NER, or a PhD in one of these areas. The reported sample does not satisfy this criterion: Figure 5 shows that 50% of the 23 respondents have less than one year of NER experience, and Figure 6 shows that 35.71% self-rate their NER experience as 'Poor' (2/5). In Section 5.1 the authors reclassify this group as experienced because of their high education level, but a general Computer Science degree is not a substitute for the stated NER expertise requirement. Since RQ1 and RQ2 are explicitly about ML experts, the aggregate results in Figure 11 and Tables 2 and 3 mix respondents who meet the stated expert definition with those who do not. The authors should either restrict the analysis to the criterion-defined experts or report a subgroup comparison demonstrating that the two groups do not differ systematically.
  2. [Section 5.4 and Section 6] There is an internal inconsistency between the sampling plan and the conclusions. Section 3 states that no representative cross section is required because the findings are not intended to be generalized, yet Section 6 concludes that the research objectives have been 'successfully achieved' and presents general statements about what 'ML experts' find important, and Section 5.4 assesses the remaining risk as 'low' despite the acknowledged low sample size. Calling the low-N threat a 'minor risk' is not justified by the evidence in the paper. The conclusions should be reframed as descriptive, hypothesis-generating findings from a convenience sample, or the authors should provide a concrete argument for why the low response count and self-selected sample do not materially affect the reported patterns.
  3. [Tables 2 and 3; Figures 10-12] The cloud-versus-local comparisons and the per-tool averages are presented as meaningful differences without any measure of uncertainty. Several cells are based on a single response (Microsoft Azure Cognitive Services and Flair in Figure 10), and even the larger per-tool cells contain only five to nine responses. For example, Table 3 reports a Cost delta of 1.79 and Table 2 reports a User interface and ease of use delta of 1.09, but with these sample sizes the deltas could easily be driven by a single respondent. The authors should report the number of responses per cell and either provide confidence intervals or a nonparametric test, or explicitly label all deltas and averages as purely descriptive with no claim of statistical reliability.
  4. [Section 5.2, Figure 12 and Table 3] The claim that 'each challenge was mentioned at least once as hindering or very hindering' supports the conclusion that requirements are project specific is too strong. In a small convenience sample, a single endorsement of each item is almost guaranteed and does not demonstrate meaningful variability. Similarly, the statement that 'reducing the time and effort required to learn new frameworks is essential' is extrapolated from an average of 2.84 on a 1-5 scale, which is between 'slightly hindering' and 'moderately hindering'. The interpretation should be calibrated to the magnitude of the observed averages and to the small number of responses per category.
minor comments (5)
  1. [Section 5.1, Figure 5] The text reports that 14.20% of participants have one to two years of NER experience, while Figure 5 shows 14.29%; the numbers should be reconciled.
  2. [Section 5.1, Figure 7] The narrative lists only Software Developer, Data Scientist, Machine Learning Engineer, Domain Expert, and Project Manager, but the figure also includes Data Engineer (8%), Researcher (8%), and Student (4%); the full set of roles should be described.
  3. [Section 5.2, Figure 10] The text says Microsoft Azure Cognitive Services has an average of 3.1 from one response, while Figure 10 reports an average of 3 from one response; the discrepancy should be corrected.
  4. [Section 5.2] The statement that OpenAI GPT-4, 'designed primarily for text generation, is highly adaptable for NER tasks' is presented without a supporting citation or evidence from the survey; either provide a reference or soften the claim.
  5. [Tables 2 and 3] The heading 'Average Priority per Selection Criteria' is imprecise; the tables report average importance ratings, not a priority ranking, and Table 3 reports hindrance ratings rather than priorities.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; the survey reports primary data, with only minor non-load-bearing self-citations.

full rationale

The paper is an expert survey: its central claims (RQ1/RQ2) are descriptive findings from 23 collected responses, such as performance being the most important selection criterion and cost/usability being salient for cloud services. These are new empirical observations, not quantities derived from the paper's own assumptions or fitted parameters. No equation in the paper reduces the reported averages or deltas to the questionnaire design by construction. The evaluation criteria were derived from prior literature and pilot testing, and the survey data then independently estimates their importance. The self-citations that appear (e.g., the authors' RecomRatio, AI4H3, CIE, FIT4NER project descriptions and earlier NER work) are motivational or contextual rather than load-bearing: the survey's conclusions would stand even if those citations were replaced by external descriptions. The most notable weakness is that 50% of respondents reported less than one year of NER experience despite the stated target-audience definition of more than three years of experience or a PhD in ML/NLP/NER, and the paper reclassifies the group as experienced on educational grounds. That is a sampling and generalizability threat, not a circular derivation; it concerns whether the respondents represent the intended population, not whether the analysis reduces to its inputs. Accordingly, the circularity score is low, reflecting only the presence of minor self-citations that do not carry the argument.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities appear in this survey paper. The central claims rest on three assumptions about participant expertise, the comprehensiveness of the listed criteria, and the adequacy of non-representative sampling. The first assumption is directly contradicted by the demographic data (50% under one year of NER experience), which weakens the generality of the findings.

assumptions (3)
  • domain assumption Survey participants are considered ML experts if they have more than three years of experience in ML, NLP, and NER, or a PhD in these areas.
    Used in the Target Audience stage to justify respondent selection, but 50% of respondents reported less than one year of NER experience, so the criterion was either not enforced or measured inconsistently.
  • domain assumption The nine listed selection criteria (performance, customization, integration, documentation, licensing, accessibility, usability, knowledge domain requirements, privacy) cover the relevant dimensions for evaluating NER tools.
    The questionnaire forces ratings on these axes; free-text fields partially mitigate omitted criteria.
  • ad hoc to paper Voluntary, non-representative sampling is sufficient because the study does not aim to generalize.
    Stated in Section 3 (Sampling Plan), yet the abstract and conclusions present findings about ML experts generally, and Section 5.4 assesses the remaining risk as low.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Survey: Understand the challenges of MachineLearning Experts using Named EntityRecognition Tools." pith.science (2026). https://pith.science/paper/XB4ZQOWM

@misc{pith2026250116112,
  author       = {Pith},
  title        = {Pith review of: Survey: Understand the challenges of MachineLearning Experts using Named EntityRecognition Tools},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XB4ZQOWM}},
  note         = {Machine review of arXiv:2501.16112}
}
read the original abstract

This paper presents a survey based on Kasunic's survey research methodology to identify the criteria used by Machine Learning (ML) experts to evaluate Named Entity Recognition (NER) tools and frameworks. Comparison and selection of NER tools and frameworks is a critical step in leveraging NER for Information Retrieval to support the development of Clinical Practice Guidelines. In addition, this study examines the main challenges faced by ML experts when choosing suitable NER tools and frameworks. Using Nunamaker's methodology, the article begins with an introduction to the topic, contextualizes the research, reviews the state-of-the-art in science and technology, and identifies challenges for an expert survey on NER tools and frameworks. This is followed by a description of the survey's design and implementation. The paper concludes with an evaluation of the survey results and the insights gained, ending with a summary and conclusions.

Figures

Figures reproduced from arXiv: 2501.16112 by the authors.

Figure 1
Figure 1. Survey Research Process by Kasunic [48] Objectives. The first stage of the Kasunic model is to establish the Research Ob￾jectives (ROs) for the survey. As Chapter 1 has already defined the RQs for this work, they can be used as a basis for establishing the ROs. The following objectives have been derived: (RO1) Identify the criteria that ML experts use to evaluate NER tools and deter￾mine the importance of each crite… view at source ↗
Figure 2
Figure 2. Age Distribution First, the demographic structure of the survey participants is analyzed. The partici￾pant group consisted predominantly of academics aged 25 to 34 years (57.14%) and 35 to 44 years (35.71%) (see [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Distribution of Academic Degrees 85.71% 7.14% 7.14% Computer Science Economics History [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Field of Study Distribution experts. Nevertheless, due to the predominant Computer Science backgrounds of the par￾ticipants, caution is warranted. The results of this survey cannot be directly generalized to domain experts and must be interpreted and evaluated accordin…
Figure 5
Figure 5. Figure 5: Distribution of NER Experience by Years Now, the experiences of the participants in the field of NER was examined [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Distribution of NER Experience Level Data Engineer Data Scientist Domain Expert Machine Learn￾ing Engineer Project Manager Researcher Software Developer Student 0% 10% 20% 30% 8% 24% 4% 16% 4% 8% 32% 4% [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Distribution of Roles in NER Projects Energy and Utilities Exploration and An￾notation of Interviews Green Ports and Shipping Healthcare and Medicine Natural Language Processing and Speech Recognition Research Studies 0% 20% 40% 5.88% 5.88% 5.88% 17.65% 47.06% 5.88% 5.…
Figure 8
Figure 8. Figure 8: Distribution of NER Project Domains utilized in the field of NLP and speech recognition (47.06%). In addition, a manifold picture emerges: NER was also applied in domains such as Healthcare and Medicine (17.65%), as well as Green Ports and Shipping and E-Commerce and R…
Figure 9
Figure 9. Figure 9: Distribution of Experience with NER Framework 0% 20% 40% 60% 80% 100% Stanford Named Entity Rec￾ognizer (avg. 2.4, 5 resp.) spaCy (avg. 2.78, 9 resp.) OpenAI GPT-4 (avg. 3.17, 6 resp.) Natural Language Toolkit (avg. 2.83, 6 resp.) Microsoft Azure Cognitive Services (av…
Figure 10
Figure 10. Figure 10: Distribution of Experience Levels per NER Framework The levels of experience of participants with various NER frameworks ( [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Priority Distribution per Selection Criteria Privacy (average 3.31) received mixed scores, signaling varying levels of concern depend￾ing on the application and deployment model. Participants also had the opportunity to specify and evaluate additional criteria in a fr…
Figure 12
Figure 12. Figure 12: Priority Distribution per Hindrance Criteria lenge was mentioned at least once as hindering or very hindering, supporting the assertion that the requirements for NER tools and frameworks are project specific. For spaCy, it was also noted that there are difficulties wi…
Figure 13
Figure 13. Figure 13: Distribution of Survey Experience [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 50 canonical work pages

  1. [1]

    Get- ting more out of biomedical documents with GATE’s full lifecycle open source text analytics

    Hamish Cunningham, Valentin Tablan, Angus Roberts, and Kalina Bontcheva. Get- ting more out of biomedical documents with GATE’s full lifecycle open source text analytics. PLoS computational biology, 9(2):e1002854, 2013. Publisher: Public Library of Science San Francisco, USA

    work page 2013

  2. [2]

    Gudivada and Kamyar Arbabifard

    Venkat N. Gudivada and Kamyar Arbabifard. Open-Source Libraries, Application Frameworks, and Workflow Systems for NLP. In Handbook of Statistics , volume 38, pages 31–50. Elsevier, 2018

    work page 2018

  3. [3]

    Mayo clinical Text Anal- ysis and Knowledge Extraction System (cTAKES): architecture, component evalu- ation and applications

    Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. Mayo clinical Text Anal- ysis and Knowledge Extraction System (cTAKES): architecture, component evalu- ation and applications. Journal of the American Medical Informatics Association , 17(5):507–513, September 2010

    work page 2010

  4. [4]

    Feature extraction method from electronic health records in Russia

    Denis Gavrilov, Alexander Gusev, Igor Korsakov, Roman Novitsky, and Larisa Serova. Feature extraction method from electronic health records in Russia. In Confer- ence of open innovations association, FRUCT , pages 497–500, 2020. Number: 26 tex.organization: FRUCT Oy

    work page 2020

  5. [5]

    Med-BERT: A Pre- Training Framework for Medical Records Named Entity Recognition

    Ning Liu, Qian Hu, Huayun Xu, Xing Xu, and Mengxin Chen. Med-BERT: A Pre- Training Framework for Medical Records Named Entity Recognition. IEEE Trans- actions on Industrial Informatics , pages 1–1, 2021

    work page 2021

  6. [6]

    Emerging Named Entity Recognition in a Medical Knowledge Management Ecosystem:

    Christian Nawroth, Felix Engel, and Matthias Hemmje. Emerging Named Entity Recognition in a Medical Knowledge Management Ecosystem:. In KEOD, pages 29– 41, Budapest, Hungary, 2020. SCITEPRESS - Science and Technology Publications

    work page 2020

  7. [7]

    The dark side of information: overload, anxiety and other paradoxes and pathologies

    David Bawden and Lyn Robinson. The dark side of information: overload, anxiety and other paradoxes and pathologies. Journal of information science , 35(2):180–191,

    work page

  8. [8]

    Clinical practice guidelines we can trust

    Earl Steinberg, Sheldon Greenfield, Dianne Miller Wolman, Michelle Mancher, Robin Graham, and others. Clinical practice guidelines we can trust . national academies press, 2011

    work page 2011

Show all 56 references
  1. [9]

    Valika, Sarah E

    Taher S. Valika, Sarah E. Maurrasse, and Lara Reichert. A Second Pandemic? Per- spective on Information Overload in the COVID-19 Era. Otolaryngology–Head and Neck Surgery, 163(5):931–933, November 2020

    2020

  2. [10]

    Towards improving clinical practice guidelines through named entity recognition: Model development and eval- uation

    Florian Freund, Philippe Tamla, and Matthias Hemmje. Towards improving clinical practice guidelines through named entity recognition: Model development and eval- uation. In 2023 31st irish conference on artificial intelligence and cognitive science (AICS), pages 1–8, 2023

    2023

  3. [11]

    Named entity recognition with a maximum en- tropy approach

    Hai Leong Chieu and Hwee Tou Ng. Named entity recognition with a maximum en- tropy approach. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4 , CONLL ’03, pages 160–163, Edmonton, Canada, May 2003. Association for Computational...

    2003

  4. [12]

    Named entity recog- nition through classifier combination

    Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. Named entity recog- nition through classifier combination. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4 , CONLL ’03, pages 168– 171, Edmonton, Canada, May 2003. A...

    2003

  5. [13]

    Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons

    Andrew McCallum and Wei Li. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In Proceed- ings of the seventh conference on Natural language learning at HLT-NAACL 2003 - , volume 4, pages 188–191, Edmonton, ...

    2003

  6. [14]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv:1802.05365 [cs] , March 2018. arXiv: 1802.05365

    2018 arXiv

  7. [15]

    Named entity recognition

    Ing Michal Konkol. Named entity recognition. PhD Thesis, PhD thesis, University of West Bohemia, 2015

    2015

  8. [16]

    A Survey on Deep Learning for Named Entity Recognition

    Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A Survey on Deep Learning for Named Entity Recognition. IEEE Transactions on Knowledge and Data Engineering , 34(1):50–70, January 2022

    2022

  9. [17]

    Classification of Machine Learning Algorithms

    Ariruna Dasgupta and Asoke Nath. Classification of Machine Learning Algorithms. Figshare, 2016

    2016

  10. [18]

    A systematic review of natural language process- ing applied to radiology reports

    Arlene Casey, Emma Davidson, Michael Poon, Hang Dong, Daniel Duma, Andreas Grivas, Claire Grover, V ´ ıctor Su´ arez-Paniagua, Richard Tobin, William Whiteley, Honghan Wu, and Beatrice Alex. A systematic review of natural language process- ing applied to radiology reports. BMC...

    2021

  11. [19]

    RATIO: Rationalizing Recommendations (RecomRatio), 2017

    Bielefeld University. RATIO: Rationalizing Recommendations (RecomRatio), 2017

    2017

  12. [20]

    Supporting Information Retrieval of Emerging Knowledge and Argumentation

    Christian Nawroth. Supporting Information Retrieval of Emerging Knowledge and Argumentation. PhD thesis, FernUniversit¨ at in Hagen, Hagen, November 2020

    2020

  13. [21]

    Andrew Taylor, Harlan M

    Irene Li, Jessica Pan, Jeremy Goldwasser, Neha Verma, Wai Pan Wong, Muhammed Yavuz Nuzumlalı, Benjamin Rosand, Yixin Li, Matthew Zhang, David Chang, R. Andrew Taylor, Harlan M. Krumholz, and Dragomir Radev. Neural Nat- ural Language Processing for Unstructured Data in Electron...

    2021 arXiv

  14. [22]

    Artificial Intelligence for Hospitals, Healthcare & Humanity (AI4H3)

    FTK. Artificial Intelligence for Hospitals, Healthcare & Humanity (AI4H3). R&D White Paper, FTK e.V. Research Institute for Telecommunications and Cooperation, Dortmund, Germany, April 2020

    2020

  15. [23]

    Cie: A cloud-based information extraction system for named entity recognition in aws, azure, and medical domain

    Philippe Tamla, Benedict Hartmann, Nhan Nguyen, Calvin Kramer, Florian Freund, and Matthias Hemmje. Cie: A cloud-based information extraction system for named entity recognition in aws, azure, and medical domain. In Frans Coenen, Ana Fred, David Aveiro, Jan Dietz, Jorge Bernar...

    2023

  16. [24]

    FIT4NER - Towards a Framework-Independent Toolkit for Named Entity Recogni- tion

    Florian Freund, Philippe Tamla, Thoralf Reis, Matthias Hemmje, and Paul Mc Kevitt. FIT4NER - Towards a Framework-Independent Toolkit for Named Entity Recogni- tion. 8th Collaborative European Research Conference (CERC 2023), Barcelona, Spain, June 9-10, 2023, 2023

    2023

  17. [25]

    A Replicable Comparison Study of NER Software: StanfordNLP, NLTK, OpenNLP, SpaCy, Gate

    Xavier Schmitt, Sylvain Kubler, Jeremy Robert, Mike Papadakis, and Yves LeTraon. A Replicable Comparison Study of NER Software: StanfordNLP, NLTK, OpenNLP, SpaCy, Gate. In 2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS) , pages ...

    2019

  18. [26]

    Bench- marking NLP Toolkits for Enterprise Application

    Kok Weiying, Duc Nghia Pham, Yasaman Eftekharypour, and Ang Jia Pheng. Bench- marking NLP Toolkits for Enterprise Application. In Abhaya C. Nayak and Alok Sharma, editors, PRICAI 2019: Trends in Artificial Intelligence , Lecture Notes in Computer Science, pages 289–294, Cham, ...

    2019

  19. [27]

    Cloud Computing – Issues, Research and Implementations

    Mladen A Vouk. Cloud Computing – Issues, Research and Implementations. Cloud Computing, page 12, 2008

    2008

  20. [28]

    Cost-Aware Elastic Cloud Provisioning for Scientific Workloads

    Ryan Chard, Kyle Chard, Kris Bubendorfer, Lukasz Lacinski, Ravi Madduri, and Ian Foster. Cost-Aware Elastic Cloud Provisioning for Scientific Workloads. In 2015 IEEE 8th International Conference on Cloud Computing , pages 971–974, New York City, NY, USA, June 2015. IEEE

    2015

  21. [29]

    Nunamaker Jr, Minder Chen, and Titus D

    Jay F. Nunamaker Jr, Minder Chen, and Titus D. M. Purdin. Systems De- velopment in Information Systems Research. Journal of Management Infor- mation Systems , 7(3):89–106, December 1990. Publisher: Routledge eprint: https://doi.org/10.1080/07421222.1990.11517898

    1990

  22. [30]

    A survey on Named Entity Recognition — datasets, tools, and methodologies

    Basra Jehangir, Saravanan Radhakrishnan, and Rahul Agarwal. A survey on Named Entity Recognition — datasets, tools, and methodologies. Natural Language Process- ing Journal , 3:100017, June 2023

    2023

  23. [31]

    Bruce Croft

    Desislava Petkova and W. Bruce Croft. Proximity-based document representation for named entity retrieval. In Proceedings of the sixteenth ACM conference on Confer- ence on information and knowledge management , pages 731–740, Lisbon Portugal, November 2007. ACM

    2007

  24. [32]

    An Approach for Ex- tracting Exact Answers to Question Answering (QA) System for English Sentences

    Raju Barskar, Gulfishan Firdose Ahmed, and Nepal Barskar. An Approach for Ex- tracting Exact Answers to Question Answering (QA) System for English Sentences. Procedia Engineering, 30:1187–1194, 2012

    2012

  25. [33]

    Transliteration normalization for information ex- traction and machine translation

    Yuval Marton and Imed Zitouni. Transliteration normalization for information ex- traction and machine translation. Journal of King Saud University-Computer and Information Sciences, 26(4):379–387, 2014. Publisher: Elsevier

    2014

  26. [34]

    Weakly labeled data augmentation for social media named entity recognition

    Juae Kim, Yejin Kim, and Sangwoo Kang. Weakly labeled data augmentation for social media named entity recognition. Expert Systems with Applications, 209:118217, December 2022

    2022

  27. [35]

    Clinical named entity recognition methods: an overview

    Naveen S Pagad and N Pradeep. Clinical named entity recognition methods: an overview. In International conference on innovative computing and communications , pages 151–165, 2022. tex.organization: Springer

    2022

  28. [36]

    Bert: Pre- training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  29. [37]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and others. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019

    2019

  30. [38]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  31. [39]

    Cloud-based system design

    Sandesh Achar. Cloud-based system design. International Journal of All Research Education and Scientific Methods (IJARESM) , 7(8):23–30, 2019

    2019

  32. [40]

    Cloud Computing: Today and Tomorrow

    Won Kim. Cloud Computing: Today and Tomorrow. CLOUD COMPUTING, 8(1):8, 2009

    2009

  33. [41]

    Towards a Privacy Compliant Cloud Architecture for Natural Language Processing Platforms:

    Matthias Blohm, Claudia Dukino, Maximilien Kintz, Monika Kochanowski, Falko Koetter, and Thomas Renner. Towards a Privacy Compliant Cloud Architecture for Natural Language Processing Platforms:. In Proceedings of the 21st International Conference on Enterprise Information Syst...

    2019

  34. [42]

    Comparing the per- formance of different NLP toolkits in formal and social media text

    Alexandre Pinto, Hugo Gon¸ calo Oliveira, and Ana Oliveira Alves. Comparing the per- formance of different NLP toolkits in formal and social media text. In Marjan Mernik, Jos´ e Paulo Leal, and Hugo Gon¸ calo Oliveira, editors,5th symposium on languages, ap- plications and tec...

    2016

  35. [43]

    Choosing an NLP Library for Analyzing Software Documentation: A Systematic Literature Review and a Series of Experiments

    Fouad Nasser A Al Omran and Christoph Treude. Choosing an NLP Library for Analyzing Software Documentation: A Systematic Literature Review and a Series of Experiments. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), pages 187–197, Buenos A...

    2017

  36. [44]

    Ensemble Named Entity Recognition (NER): Evaluating NER Tools in the Identification of Place Names in Historical Corpora

    Miguel Won, Patricia Murrieta-Flores, and Bruno Martins. Ensemble Named Entity Recognition (NER): Evaluating NER Tools in the Identification of Place Names in Historical Corpora. Frontiers in Digital Humanities , 5:2, March 2018

    2018

  37. [45]

    Comparing open ara- bic named entity recognition tools, 2022

    Abdullah Aldumaykhi, Saad Otai, and Abdulkareem Alsudais. Comparing open ara- bic named entity recognition tools, 2022

    2022

  38. [46]

    Soft- ware Engineering for Machine Learning: A Case Study

    Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. Soft- ware Engineering for Machine Learning: A Case Study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: S...

    2019

  39. [47]

    Important Experimentation Characteristics: An Expert Survey

    Florian Auer and Michael Felderer. Important Experimentation Characteristics: An Expert Survey. In Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) , pages 1–6, Bari Italy, October 2021. ACM

    2021

  40. [48]

    Designing an Effective Survey

    Mark Kasunic. Designing an Effective Survey . Citeseer, September 2005

    2005

  41. [49]

    An expert survey to assess the current status and future challenges of energy system analysis

    Fabian Scheller, Frauke Wiese, Jann Michael Weinand, Dominik Franjo Dominkovi´ c, and Russell McKenna. An expert survey to assess the current status and future challenges of energy system analysis. Smart Energy, 4:100057, 2021

    2021

  42. [50]

    Relating verification and validation methods to software product quality characteristics: Re- sults of an expert survey

    Isela Mendoza, Marcos Kalinowski, U´ everton Souza, and Michael Felderer. Relating verification and validation methods to software product quality characteristics: Re- sults of an expert survey. In Dietmar Winkler, Stefan Biffl, and Johannes Bergsmann, editors, Software qualit...

    2019

  43. [51]

    Introduction to Transformers for NLP: With the Hugging Face Library and Models to Solve Problems

    Shashank Mohan Jain. Introduction to Transformers for NLP: With the Hugging Face Library and Models to Solve Problems . Apress, Berkeley, CA, 2022

    2022

  44. [52]

    GPT-4 Technical Report, March 2024

    OpenAI et al. GPT-4 Technical Report, March 2024

    2024

  45. [53]

    Comprehensive Overview of Named Entity Recognition: Models, Domain-Specific Applications and Challenges, September 2023

    Kalyani Pakhale. Comprehensive Overview of Named Entity Recognition: Models, Domain-Specific Applications and Challenges, September 2023. arXiv:2309.14084 [cs]

    2023 arXiv

  46. [54]

    Kurdi, Safwat Hamad, and Amal Khalifa

    Heba A. Kurdi, Safwat Hamad, and Amal Khalifa. Towards a Friendly User Interface on the Cloud. In David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Alfred Kobsa, Friedemann Mattern, John C. Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Steffen,...

    2014

  47. [55]

    FinGPT: Open-Source Financial Large Language Models, June 2023

    Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. FinGPT: Open-Source Financial Large Language Models, June 2023. arXiv:2306.06031 [q-fin]

    2023

  48. [2009]

    Publisher: Sage Publications Sage UK: London, England

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.