REVIEW 23 cited by
LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Deductive coding is a widely used qualitative research method for determining the prevalence of themes across documents. While useful, deductive coding is often burdensome and time consuming since it requires researchers to read, interpret, and reliably categorize a large body of unstructured text documents. Large language models (LLMs), like ChatGPT, are a class of quickly evolving AI tools that can perform a range of natural language processing and reasoning tasks. In this study, we explore the use of LLMs to reduce the time it takes for deductive coding while retaining the flexibility of a traditional content analysis. We outline the proposed approach, called LLM-assisted content analysis (LACA), along with an in-depth case study using GPT-3.5 for LACA on a publicly available deductive coding data set. Additionally, we conduct an empirical benchmark using LACA on 4 publicly available data sets to assess the broader question of how well GPT-3.5 performs across a range of deductive coding tasks. Overall, we find that GPT-3.5 can often perform deductive coding at levels of agreement comparable to human coders. Additionally, we demonstrate that LACA can help refine prompts for deductive coding, identify codes for which an LLM is randomly guessing, and help assess when to use LLMs vs. human coders for deductive coding. We conclude with several implications for future practice of deductive coding and related research methods.
Forward citations
Cited by 23 Pith papers
-
My Favorite Streamer is an LLM: Discovering, Bonding, and Co-Creating in AI VTuber Fandom
Fans of AI VTuber Neuro-sama treat her as a consistent persona, not a human mimic, and use paid SuperChats to co-create stream content, creating a distinct participatory fan economy.
-
Design-Based Supervised Learning with Noisy Human Labels
A nested design-based correction that combines noisy human audit labels with partial expert adjudication yields unbiased downstream estimates and better efficiency than using only adjudicated labels.
-
Commenting with Copilot: A Taxonomy and Multi-Year Analysis of Student Code-Generation Specifications
In four years of Copilot tasks, students wrote mostly natural-language What comments, used more How comments for procedural constructs, and focused effort on verifying output rather than rewriting comments.
-
The Role of Partisan Culture in Mental Health Language Online
Partisan culture is associated with measurable differences in how Republican and Democrat Reddit users express distress, with Democrats using more clinical and polarization-related language and Republicans using more ...
-
SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models
SocialMaze is a six-task benchmark that claims to evaluate LLM social reasoning along deep reasoning, dynamic interaction, and information uncertainty dimensions.
-
Simulating Ethics: Using LLM Debate Panels to Model Deliberation on Medical Dilemmas
Two AI ethics debates with differently composed panels reached the same policy recommendation but through different arguments and coalitions, showing that panel membership can shift reasoning even when the facts are fixed.
-
ChatCollab: Exploring Collaboration Between Humans and AI Agents in Software Teams
ChatCollab lets humans and AI agents work side-by-side in software teams and offers a way to measure how those teams collaborate.
-
An Empirical Examination of the Evaluative AI Framework
A pre-registered experiment found that an AI providing only pro and con evidence, without recommendations, did not improve decision performance and was used shallowly by participants.
-
A Computational Ethical Framework for Financial Digital Phenotyping for Mental Health
Ethical rules for financial digital phenotyping can be written as deontic temporal constraints whose violations Z3 proves unsatisfiable inside the formal model.
-
Identifying Implicit Bias in LLM-based Chat AI Toward People with Intellectual Disabilities
Across 25,000 stories from five LLMs, an LLM judge rated stories mentioning intellectual disabilities as more infantile, paternalistic, dependent, and inspirational than stories without the label.
-
EchoAid: Enhancing Livestream Shopping Accessibility for the DHH Community
EchoAid combines speech-to-text, LLM summarization, and rapid serial visual presentation to help deaf and hard of hearing users follow livestream shopping shows; user studies show lower cognitive load and modestly bet...
-
From Assistance to Autonomy -- A Researcher Study on the Potential of AI Support for Qualitative Data Analysis
Interviews with 15 HCI researchers show openness to AI in qualitative data analysis under conditions of privacy, control, and reliability, leading to a framework of AI involvement levels from minimal to high.
-
Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation
An LLM code optimizer using control-flow-graph differences and retrieved examples reports 7.3% average runtime reduction on 116 C++ programs versus zero-shot GPT-4o.
-
LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning
LeMo reduces long-context fine-tuning memory by eliminating low-informativeness tokens, predicting sparsity patterns, and optimizing kernels, while keeping perplexity close to LoRA.
-
Interpreting Language Reward Models via Contrastive Explanations
Reward model preferences can be explained by generating counterfactual and semifactual answer variations along 15 hand-picked evaluation attributes and measuring which attribute changes flip the model's preference.
-
Conversational AI for Rapid Scientific Prototyping: A Case Study on ESA's ELOPE Competition
One engineer paired with ChatGPT and reached second place in ESA's ELOPE competition in about one week of work; the paper draws best-practice lessons from that experience.
-
A Confidence-Diversity Framework for Calibrating AI Judgement in Accessible Qualitative Coding Tasks
A risk score built from confidence and vote entropy is proposed for triaging LLM qualitative coding, but the central R-squared=0.979 result is inflated by the definitions of agreement and diversity.
-
From Inductive to Deductive: LLMs-Based Qualitative Data Analysis in Requirements Engineering
GPT-4 labels software requirements with substantial agreement to human analysts (Cohen's Kappa up to 0.738) when given detailed few-shot prompts, while zero-shot performance is only moderate.
-
Concept Navigation and Classification via Open-Source Large Language Model Processing
A pipeline combining LLM summarization, iterative category generation, and human-in-the-loop refinement achieves frame and topic classification accuracy comparable to human coders on three text corpora.
-
A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions
A review that organizes LLM uncertainty quantification into token-level, self-verbalized, semantic-similarity, and mechanistic interpretability categories.
-
Scout: Leveraging Large Language Models for Rapid Digital Evidence Discovery
Scout applies off-the-shelf LLMs and vision models to triage digital evidence, but only anecdotal examples are shown and accuracy is withheld.
-
Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.
-
A Framework for LLM-powered Design Assistants
LLMs are organized into a three-modality framework for design assistance, but no evidence is provided that the framework works.
Discussion (0). Continue with ORCID to comment.