REVIEW 15 cited by
Automated Annotation with Generative AI Requires Validation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Generative large language models (LLMs) can be a powerful tool for augmenting text annotation procedures, but their performance varies across annotation tasks due to prompt quality, text data idiosyncrasies, and conceptual difficulty. Because these challenges will persist even as LLM technology improves, we argue that any automated annotation process using an LLM must validate the LLM's performance against labels generated by humans. To this end, we outline a workflow to harness the annotation potential of LLMs in a principled, efficient way. Using GPT-4, we validate this approach by replicating 27 annotation tasks across 11 datasets from recent social science articles in high-impact journals. We find that LLM performance for text annotation is promising but highly contingent on both the dataset and the type of annotation task, which reinforces the necessity to validate on a task-by-task basis. We make available easy-to-use software designed to implement our workflow and streamline the deployment of LLMs for automated annotation.
Forward citations
Cited by 15 Pith papers
-
Talking Politics with Artificial Intelligence
Large-scale analysis of AI conversations indicates they function primarily as practical intermediaries for political tasks rather than arenas for public expression, with increased expressiveness after major events.
-
The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel
A panel of nine LLMs achieves Krippendorff's alpha of 0.86 for political position measurement in data-sparse regions, with added axis definitions improving agreement and disagreements revealing interpretive issues.
-
LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification
Fine-tuned BERTimbau-LoRA achieves 87.6% accuracy and 0.87 macro-F1 on LegalBench-BR, outperforming commercial LLMs by 22-28 points and eliminating their systematic bias toward civil law on Brazilian legal classification.
-
Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection
LLM annotation can replace human labels for hostility detection with comparable F1 at much lower cost, but active learning adds little value and error structures differ systematically.
-
Guidelines for Empirical Studies in Software Engineering involving Large Language Models
The paper delivers a taxonomy of seven LLM study types in software engineering along with eight guidelines that separate mandatory requirements from recommended practices to address reproducibility challenges.
-
Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection
With a two-question interface, full-pool LLM annotation (GPT-5.2 or Qwen3.5) beats human-supervised classifiers for anti-immigrant hostility detection at ~1/10 cost; AL does not reliably beat random sampling.
-
Auditing Differential Visibility of Political Content on TikTok
Account-level analysis finds no evidence of moderate-to-large reach suppression on TikTok for three political topics; an apparent pooled gap is a statistical artifact, while oppositional content earns more engagement ...
-
Grounded Event Extraction from SEC 8-K Filings with a Fine-Grained Taxonomy
Schema-constrained, quote-grounded LLM extraction plus a second-pass quality score yields 601k auditable 8-K event tags whose precision and market reactions both improve with the score.
-
Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
Grain calibration decomposes theoretical constructs into clause-level components, tests each with extractive evidence, and combines results through explicit theory-derived rules to validate LLM coding beyond agreement...
-
What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Media
Introduces SopriBench benchmark and Argus agentic framework for user-level multimodal privacy leakage inference, reporting 0.55 PES with 25% improvement over baselines.
-
Guidelines for Empirical Studies in Software Engineering involving Large Language Models
A group of 22 researchers proposes seven study types and eight guidelines for empirical software engineering studies involving LLMs to enhance reproducibility and replicability.
-
A Durability and Cross-Language Transfer Benchmark for a Validated Teaching-Feedback Classification Protocol
The teaching-feedback classification protocol remains durable across three representation generations and transfers to English sentiment, so model choice is a deployment decision.
-
Talking Politics with Artificial Intelligence
Political content appears in 3.9% of AI conversations, mostly for information and drafting rather than opinions, with U.S. users showing increased stance-taking and affect after the 2024 election result call.
-
How Much Does Persuasion Strategy Matter? LLM-Annotated Evidence from Charitable Donation Dialogues
LLM annotation of donation dialogues shows persuasion strategy categories explain only 1.5% of outcome variance, with guilt induction linked to 23 percentage point lower donation rates that replicate across models.
-
LLM Predictive Scoring and Validation: Inferring Experience Ratings from Unstructured Text
GPT-4.1 predictions from fan text match self-reported experience ratings within one point 67% of the time but are biased low by one point, interpreted as measuring salient moments versus integrated overall judgment.
Discussion (0). Sign in to comment.