Pith. sign in

REVIEW 8 cited by

Can LLMs Replace Manual Annotation of Software Engineering Artifacts?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.05534 v2 pith:FECKZHZO submitted 2024-08-10 cs.SE cs.HCcs.LG

Toufique Ahmed , Premkumar Devanbu , Christoph Treude , Michael Pradel This is my paper · ORCID
classification cs.SEcs.HCcs.LG
keywords llmsannotationengineeringevaluationshumanhuman-subjectsoftwarestudies
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Experimental evaluations of software engineering innovations, e.g., tools and processes, often include human-subject studies as a component of a multi-pronged strategy to obtain greater generalizability of the findings. However, human-subject studies in our field are challenging, due to the cost and difficulty of finding and employing suitable subjects, ideally, professional programmers with varying degrees of experience. Meanwhile, large language models (LLMs) have recently started to demonstrate human-level performance in several areas. This paper explores the possibility of substituting costly human subjects with much cheaper LLM queries in evaluations of code and code-related artifacts. We study this idea by applying six state-of-the-art LLMs to ten annotation tasks from five datasets created by prior work, such as judging the accuracy of a natural language summary of a method or deciding whether a code change fixes a static analysis warning. Our results show that replacing some human annotation effort with LLMs can produce inter-rater agreements equal or close to human-rater agreement. To help decide when and how to use LLMs in human-subject studies, we propose model-model agreement as a predictor of whether a given task is suitable for LLMs at all, and model confidence as a means to select specific samples where LLMs can safely replace human annotators. Overall, our work is the first step toward mixed human-LLM evaluations in software engineering.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Delving into Instance-Dependent Label Noise in Graph Data: A Comprehensive Study and Benchmark

    cs.LG 2025-06 conditional novelty 7.0 of 10

    BeGIN provides 10 graph datasets with six label-noise types and a broad evaluation, showing that LLM-simulated noise hurts GNNs more than uniform or pairwise noise.

  2. ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering

    cs.SE 2026-08 conditional novelty 6.0 of 10

    ECLAIR integrates LLMs into a human-in-the-loop, causal-inference-based scientific workflow for software engineering, and a pilot study finds small negative causal effects of certain prompt designs on code generation ...

  3. Evaluating the Use of LLMs for Documentation to Code Traceability

    cs.SE 2025-06 conditional novelty 6.0 of 10

    LLMs identify documentation-to-code trace links with F1 up to 80.4%, outperforming TF-IDF, BM25, and CodeBERT, but their explanations and chain reconstructions need human oversight.

  4. Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement

    cs.SE 2025-05 conditional novelty 6.0 of 10

    AlignMind, a multi-agent system with theory-of-mind helpers, refines software requirements through multi-round dialogue, outperforming a direct-prompt baseline in LLM-judged quality and lexical richness, at high token...

  5. The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A new statistical test, the alt-test, lets researchers decide from about 50 to 100 double-annotated examples whether an LLM can replace human annotators.

  6. BitsAI-CR: Automated Code Review via LLM in Practice

    cs.SE 2025-01 conditional novelty 5.0 of 10

    An industrial LLM-based code review system with a two-stage generate-and-filter pipeline and a data flywheel reached 75% precision and a 26.7% developer-action rate on Go code at ByteDance.

  7. Large Language Models as Robust Data Generators in Software Analytics: Are We There Yet?

    cs.SE 2024-11 conditional novelty 5.0 of 10

    Pre-trained models fine-tuned on LLM-generated data are less robust to adversarial attacks than models fine-tuned on human-written data in three software analytics tasks.

  8. Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants

    cs.SE 2025-02 unverdicted novelty 4.0 of 10

    A position paper proposing that combining simulated users and LLM-as-a-Judge can meet the requirements for human-centered automatic evaluation of conversational SE assistants, without empirical validation.

Pith tools