A panel of nine LLMs achieves Krippendorff's alpha of 0.86 for political position measurement in data-sparse regions, with added axis definitions improving agreement and disagreements revealing interpretive issues.
Moore, Daniel M
10 Pith papers cite this work, alongside 35 external citations. Polarity classification is still indexing.
10
Pith papers citing it
35
external citations · Pith
abstract
Generative large language models (LLMs) can be a powerful tool for augmenting text annotation procedures, but their performance varies across annotation tasks due to prompt quality, text data idiosyncrasies, and conceptual difficulty. Because these challenges will persist even as LLM technology improves, we argue that any automated annotation process using an LLM must validate the LLM's performance against labels generated by humans. To this end, we outline a workflow to harness the annotation potential of LLMs in a principled, efficient way. Using GPT-4, we validate this approach by replicating 27 annotation tasks across 11 datasets from recent social science articles in high-impact journals. We find that LLM performance for text annotation is promising but highly contingent on both the dataset and the type of annotation task, which reinforces the necessity to validate on a task-by-task basis. We make available easy-to-use software designed to implement our workflow and streamline the deployment of LLMs for automated annotation.
citation-role summary
background 2
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
Fine-tuned BERTimbau-LoRA achieves 87.6% accuracy and 0.87 macro-F1 on LegalBench-BR, outperforming commercial LLMs by 22-28 points and eliminating their systematic bias toward civil law on Brazilian legal classification.
The paper delivers a taxonomy of seven LLM study types in software engineering along with eight guidelines that separate mandatory requirements from recommended practices to address reproducibility challenges.
With a two-question interface, full-pool LLM annotation (GPT-5.2 or Qwen3.5) beats human-supervised classifiers for anti-immigrant hostility detection at ~1/10 cost; AL does not reliably beat random sampling.
Schema-constrained, quote-grounded LLM extraction plus a second-pass quality score yields 601k auditable 8-K event tags whose precision and market reactions both improve with the score.
Grain calibration decomposes theoretical constructs into clause-level components, tests each with extractive evidence, and combines results through explicit theory-derived rules to validate LLM coding beyond agreement with human annotators.
Introduces SopriBench benchmark and Argus agentic framework for user-level multimodal privacy leakage inference, reporting 0.55 PES with 25% improvement over baselines.
Political content appears in 3.9% of AI conversations, mostly for information and drafting rather than opinions, with U.S. users showing increased stance-taking and affect after the 2024 election result call.
LLM annotation of donation dialogues shows persuasion strategy categories explain only 1.5% of outcome variance, with guilt induction linked to 23 percentage point lower donation rates that replicate across models.
GPT-4.1 predictions from fan text match self-reported experience ratings within one point 67% of the time but are biased low by one point, interpreted as measuring salient moments versus integrated overall judgment.
citing papers explorer
-
The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel
A panel of nine LLMs achieves Krippendorff's alpha of 0.86 for political position measurement in data-sparse regions, with added axis definitions improving agreement and disagreements revealing interpretive issues.
-
LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification
Fine-tuned BERTimbau-LoRA achieves 87.6% accuracy and 0.87 macro-F1 on LegalBench-BR, outperforming commercial LLMs by 22-28 points and eliminating their systematic bias toward civil law on Brazilian legal classification.
-
Guidelines for Empirical Studies in Software Engineering involving Large Language Models
The paper delivers a taxonomy of seven LLM study types in software engineering along with eight guidelines that separate mandatory requirements from recommended practices to address reproducibility challenges.
-
Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection
With a two-question interface, full-pool LLM annotation (GPT-5.2 or Qwen3.5) beats human-supervised classifiers for anti-immigrant hostility detection at ~1/10 cost; AL does not reliably beat random sampling.
-
Grounded Event Extraction from SEC 8-K Filings with a Fine-Grained Taxonomy
Schema-constrained, quote-grounded LLM extraction plus a second-pass quality score yields 601k auditable 8-K event tags whose precision and market reactions both improve with the score.
-
Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
Grain calibration decomposes theoretical constructs into clause-level components, tests each with extractive evidence, and combines results through explicit theory-derived rules to validate LLM coding beyond agreement with human annotators.
-
What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Media
Introduces SopriBench benchmark and Argus agentic framework for user-level multimodal privacy leakage inference, reporting 0.55 PES with 25% improvement over baselines.
-
Talking Politics with Artificial Intelligence
Political content appears in 3.9% of AI conversations, mostly for information and drafting rather than opinions, with U.S. users showing increased stance-taking and affect after the 2024 election result call.
-
How Much Does Persuasion Strategy Matter? LLM-Annotated Evidence from Charitable Donation Dialogues
LLM annotation of donation dialogues shows persuasion strategy categories explain only 1.5% of outcome variance, with guilt induction linked to 23 percentage point lower donation rates that replicate across models.
-
LLM Predictive Scoring and Validation: Inferring Experience Ratings from Unstructured Text
GPT-4.1 predictions from fan text match self-reported experience ratings within one point 67% of the time but are biased low by one point, interpreted as measuring salient moments versus integrated overall judgment.