Pith. sign in

REVIEW 9 cited by

Can Large Language Models Transform Computational Social Science?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.03514 v3 pith:ZBF2VJ77 submitted 2023-04-12 cs.CL cs.LG

Caleb Ziems , William Held , Omar Shaikh , Jiaao Chen , Zhehao Zhang , Diyi Yang This is my paper · ORCID
classification cs.CLcs.LG
keywords llmslanguagemodelssocialtaskszero-shotpipelinescience
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large Language Models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify and explain social phenomena like persuasiveness and political ideology, then LLMs could augment the Computational Social Science (CSS) pipeline in important ways. This work provides a road map for using LLMs as CSS tools. Towards this end, we contribute a set of prompting best practices and an extensive evaluation pipeline to measure the zero-shot performance of 13 language models on 25 representative English CSS benchmarks. On taxonomic labeling tasks (classification), LLMs fail to outperform the best fine-tuned models but still achieve fair levels of agreement with humans. On free-form coding tasks (generation), LLMs produce explanations that often exceed the quality of crowdworkers' gold references. We conclude that the performance of today's LLMs can augment the CSS research pipeline in two ways: (1) serving as zero-shot data annotators on human annotation teams, and (2) bootstrapping challenging creative generation tasks (e.g., explaining the underlying attributes of a text). In summary, LLMs are posed to meaningfully participate in social science analysis in partnership with humans.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  2. LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A benchmark for LLM agents in partially observable joint decision-making reveals that deliberation challenges current models but can enable reflection and error correction.

  3. Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch

    cs.CL 2025-07 conditional novelty 6.0 of 10

    The paper presents a Dutch adaptation of the CrowS-Pairs bias benchmark and reports bias scores for seven masked and two autoregressive language models across nine demographic categories.

  4. The Role of Partisan Culture in Mental Health Language Online

    cs.HC 2025-06 conditional novelty 6.0 of 10

    Partisan culture is associated with measurable differences in how Republican and Democrat Reddit users express distress, with Democrats using more clinical and polarization-related language and Republicans using more ...

  5. LEAP: LLM-powered End-to-end Automatic Library for Processing Social Science Queries on Unstructured Data

    cs.DB 2025-01 conditional novelty 6.0 of 10

    LEAP, an LLM-based library, automatically selects ML functions and writes SQL-like code to answer 92% of 120 social science queries over unstructured data on the first attempt, and 100% within three attempts.

  6. Generative Adversarial Reviews: When LLMs Become the Critic

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A new LLM-agent framework, GAR, generates peer reviews from a graph representation of manuscripts and predicts conference acceptance decisions, reportedly matching or exceeding human reviewer performance.

  7. From Assistance to Autonomy -- A Researcher Study on the Potential of AI Support for Qualitative Data Analysis

    cs.CY 2025-01 conditional novelty 5.0 of 10

    Interviews with 15 HCI researchers show openness to AI in qualitative data analysis under conditions of privacy, control, and reliability, leading to a framework of AI involvement levels from minimal to high.

  8. CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation

    cs.CL 2025-06 conditional novelty 4.0 of 10

    CCRS is a zero-shot LLM-as-a-judge framework whose five metrics discriminate between RAG systems on BioASQ with comparable or better power than RAGChecker at lower compute.

  9. Artificially intelligent agents in the social and behavioral sciences: A history and outlook

    cs.AI 2025-10 conditional novelty 2.0 of 10

    AI and social science have co-evolved for 75 years through rapid technological adoption and slower scientific consolidation, with direct human-focused AI studies still scarce.

Pith tools