pith. sign in

arxiv: 2303.16854 · v2 · pith:SIX2PNGLnew · submitted 2023-03-29 · 💻 cs.CL

AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators

Xingwei He , Zhenghao Lin , Yeyun Gong , A-Long Jin , Hang Zhang , Chen Lin , Jian Jiao , Siu Ming Yiu
show 2 more authors
Nan Duan Weizhu Chen
This is my paper
classification 💻 cs.CL
keywords modelsannollmdatallmscrowdsourceddatasetfirstlanguage
0
0 comments X
read the original abstract

Many natural language processing (NLP) tasks rely on labeled data to train machine learning models with high performance. However, data annotation is time-consuming and expensive, especially when the task involves a large amount of data or requires specialized domains. Recently, GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. In this paper, we first claim that large language models (LLMs), such as GPT-3.5, can serve as an excellent crowdsourced annotator when provided with sufficient guidance and demonstrated examples. Accordingly, we propose AnnoLLM, an annotation system powered by LLMs, which adopts a two-step approach, explain-then-annotate. Concretely, we first prompt LLMs to provide explanations for why the specific ground truth answer/label was assigned for a given example. Then, we construct the few-shot chain-of-thought prompt with the self-generated explanation and employ it to annotate the unlabeled data with LLMs. Our experiment results on three tasks, including user input and keyword relevance assessment, BoolQ, and WiC, demonstrate that AnnoLLM surpasses or performs on par with crowdsourced annotators. Furthermore, we build the first conversation-based information retrieval dataset employing AnnoLLM. This dataset is designed to facilitate the development of retrieval models capable of retrieving pertinent documents for conversational text. Human evaluation has validated the dataset's high quality.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Evaluating Very Long-Term Conversational Memory of LLM Agents

    cs.CL 2024-02 unverdicted novelty 8.0

    Creates LoCoMo benchmark dataset for very long-term LLM conversational memory and shows current models struggle with lengthy dialogues and long-range temporal dynamics.

  2. Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation

    cs.CL 2025-11 unverdicted novelty 7.0

    LLMs produce stable cognitive distortion labels that improve downstream model performance, paired with a kappa-based framework for dataset-agnostic evaluation in subjective NLP tasks.

  3. WizardLM: Empowering large pre-trained language models to follow complex instructions

    cs.CL 2023-04 conditional novelty 7.0

    WizardLM uses LLM-driven iterative rewriting to generate complex instruction data and fine-tunes LLaMA to reach over 90% of ChatGPT capacity on 17 of 29 evaluated skills.

  4. A Scalable Tool for Measuring Manner and Result Verbs in Developmental Language Research

    cs.CL 2026-05 conditional novelty 6.0

    A RoBERTa classifier trained on LLM-generated manner/result verb annotations from extended VerbNet data reaches up to 89.6% accuracy on held-out gold-standard sets.

  5. Context-Aware Spear Phishing: Generative AI-Enabled Attacks Against Individuals via Public Social Media Data

    cs.CR 2026-05 conditional novelty 6.0

    Generative AI enables scalable, context-aware spear phishing by extracting profiles from public social media, producing emails that outperform real-world phishing samples in personalization and lower recipient suspicion.

  6. Cross-Cultural Transfer of Emoji Semantics and Sentiment in Financial Social Media

    cs.CL 2026-05 unverdicted novelty 6.0

    Emojis exhibit stable semantics and sentiment polarity in financial social media across languages and assets, reducing gaps in zero-shot sentiment transfer when incorporated into models.

  7. Dynamic Emotion and Personality Profiling for Multimodal Deception Detection

    cs.CL 2026-04 unverdicted novelty 6.0

    A new dataset DDEP and reliability-weighted fusion model Rel-DDEP jointly detect deception, emotion, and personality from multimodal data, reporting F1 gains of 2.53%, 2.66%, and 9.30% over baselines.

  8. Characterizing initial human-AI proof formalization workflows

    cs.AI 2026-06 unverdicted novelty 5.0

    A controlled user study and qualitative survey find that AI assistance raises formalization accuracy for math proofs, with users flexibly combining multiple tools while retaining oversight.

  9. Structured Exploration and Exploitation of Label Functions for Automated Data Annotation

    cs.LG 2026-03 unverdicted novelty 5.0

    EXPONA improves automated data labeling by exploring multi-level label functions and applying reliability filters, achieving up to 98.9% coverage and 46% gains in downstream weighted F1 on eleven datasets.

  10. No Data? No Problem: Synthesizing Security Graphs for Better Intrusion Detection

    cs.CR 2025-06 unverdicted novelty 5.0

    PROVSYN synthesizes high-fidelity security provenance graphs via graph generation and LLMs to augment imbalanced datasets, improving downstream APT detection accuracy by up to 38% on benchmarks.

  11. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

    cs.CL 2024-12 accept novelty 3.0

    A survey that organizes LLMs-as-judges research into functionality, methodology, applications, meta-evaluation, and limitations.

  12. A Survey on Knowledge Distillation of Large Language Models

    cs.CL 2024-02 accept novelty 3.0

    A comprehensive survey of knowledge distillation for LLMs structured around algorithms, skill enhancement, and vertical applications, highlighting data augmentation as a key enabler.