Pith. sign in

REVIEW 2 major objections 14 references

Under a two-question annotation interface, LLM labels at full-corpus scale produce stronger hostility detectors than human labels at about one-tenth the cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 20:33 UTC pith:FBRGX74K

load-bearing objection Solid, careful empirical result: with a two-question interface, full LLM labeling at scale beats full human supervision on this hostility task at ~1/10 cost; AL adds little in a prefiltered pool. the 2 major comments →

arxiv 2604.13899 v4 pith:FBRGX74K submitted 2026-04-15 cs.CL cs.AI

Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection

Ahmad Dawar Hakimi , Lea Hirlimann , Isabelle Augenstein , Hinrich Sch\"utze This is my paper
classification cs.CL cs.AI
keywords active learningLLM annotationhate speech detectionanti-immigrant hostilityGerman TikToksoft labelsannotation interfacecost-effectiveness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether instruction-tuned language models can replace human annotators when training classifiers for a hard, subjective task: detecting anti-immigrant hostility in German political TikTok comments. The authors build a large new corpus, label thousands of comments with both crowdworkers and two frontier LLMs, and train the same lightweight classifiers under matched human and LLM conditions, including active learning and full-pool labeling. Their central result is that once the LLM is given the same two-question interface humans use, labeling the entire relevant pool yields better classifiers than full human supervision, at roughly one-tenth the money, for both a closed-source and an open-weight model. Active learning does not reliably beat random sampling in this pre-filtered pool. Error patterns still differ by model and prompt, so the practical question shifts from whether to use LLM labels to which model, which interface, and what shape of pool.

Core claim

Under a two-question interface that mirrors the human task, full-pool LLM annotation of about 26,000 immigration-relevant German TikTok comments produces classifiers that outperform the same models trained on 3,800 human labels, at roughly one-tenth the annotation cost, for both GPT-5.2 and Qwen3.5-122B. The gain is robust under soft-label evaluation that respects annotator disagreement, and it disappears when a holistic single-prompt interface is used instead. Active learning acquisition strategies give no reliable edge over random sampling under either LLM annotator.

What carries the argument

The two-question (decomposed) annotation interface: Q1 asks whether the comment refers to immigrants or immigration policy; Q2 asks whether those groups are portrayed negatively. The final anti-immigrant label is Q1-yes and Q2-yes. Matching this interface for humans and LLMs is what unlocks the scale advantage; a holistic single-prompt baseline only ties human supervision.

Load-bearing premise

The paper treats majority-vote and soft-label human crowdworker labels on the gold set as the right yardstick for declaring LLM-trained classifiers better, even while documenting systematic topical disagreements where every LLM over-flags some policy themes.

What would settle it

Retrain and re-evaluate the same full-pool LLM and human conditions against expert-adjudicated labels on the contested Border Control and metaphor-heavy clusters, or on a later out-of-distribution time slice; if the 26K LLM advantage vanishes or reverses under that gold standard, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For resource-constrained research on subjective German political text, full-corpus LLM labeling under a decomposed interface can replace a multi-hundred-dollar human study for training labels.
  • Active learning is unlikely to justify its overhead once the pool has already been topic-prefiltered for relevance.
  • Practitioners must choose the LLM and prompt carefully: only the GPT two-question setup yields near-human false-positive/false-negative balance; other variants over-flag border-control and economic-competition discourse.
  • Moderation pipelines that triage low-confidence predictions will surface fewer reviewable errors under LLM-supervised classifiers, because those models make more high-confidence mistakes.
  • Humans remain essential as gold-standard evaluators; the training-label question becomes which LLM, which interface, and what pool shape.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same decomposed-interface gain may transfer to other policy-versus-group hostility tasks where a holistic prompt collapses topic and valence into one co-occurrence shortcut.
  • If open-weight models keep closing the quality gap at local-hardware cost, the remaining bottleneck will be interface design and topical calibration rather than API access.
  • Prefiltering that concentrates relevant content may systematically erase the diversity that diversity-based active learning needs, suggesting AL comparisons should report pool enrichment as a design variable.
  • High-confidence false positives on border and welfare discourse imply that unsupervised deployment of these classifiers could suppress legitimate policy speech even when aggregate F1 looks strong.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper asks whether LLM labels can replace human labels in active learning for anti-immigrant hostility detection, and whether AL remains useful when full-corpus LLM labeling is cheap. On a new corpus of 277,902 German political TikTok comments (25,974 LLM-labeled; 5,000 human-annotated), it compares seven conditions (AL/random/full × human/LLM, plus FULL-LLM-26K) across four encoders and 10 seeds. Under a two-question (Q1 reference / Q2 valence) interface matching the human task, FULL-LLM-26K with GPT-5.2 and with Qwen3.5-122B outperforms FULL-HUMAN on macro-F1 and F1-Anti at roughly one-tenth the cost; a holistic single-prompt ablation only ties. AL does not reliably beat random sampling under either annotator. Soft-label evaluation (Beta-Binomial, Dawid-Skene) preserves the ranking. Error structure differs: only GPT-D yields near-human FP/FN balance; other variants over-flag border-control and economic-competition discourse. Dataset and code are released.

Significance. If the result holds, it is a practically important finding for subjective NLP: under a matched decomposed interface, full-pool LLM annotation can dominate human-supervised training on cost and F1, for both closed and open-weight models, with soft-label robustness and a clear interface ablation. Strengths include the matched seven-condition design, four encoders × 10 seeds, four acquisition functions, prefilter validation, soft-label posteriors, interface and reasoning ablations, topical BERTopic analysis, and cost accounting with released data/code. The AL-null result is carefully scoped to a pre-enriched pool. The work is a solid empirical contribution to LLM-as-annotator and AL practice, with actionable caveats on error structure and evaluation against crowd gold.

major comments (2)
  1. The central claim is evaluated exclusively against majority-vote (and BB/DS soft) human crowd labels on the 1,200-item gold set (§3.4, §3.7, Table 4, App. F). The paper itself documents systematic topical divergence (Border Control over-flagged by every LLM variant; §4.5–4.6, Table 19) and states in Limitations that human-gold evaluation structurally disadvantages divergent boundaries. Soft-label checks still favor FULL-LLM-26K, so the reported ordering is internally consistent, but the claim that LLM-at-scale is 'better' remains relative to this yardstick. A short expert adjudication (or dual-evaluation) on the contested clusters would substantially strengthen the deployment interpretation without changing the experimental design.
  2. The AL-versus-random null (§4.3, Table 20, RQ2) is conditioned on a Llama-prefiltered pool that already concentrates immigration-relevant content (§3.3, §5.4). The authors note this, but the abstract and title frame the question as whether AL remains necessary when corpora can be cheaply labeled. The manuscript should state more prominently (abstract/conclusion) that the AL-null is for pre-enriched pools; without an unfiltered-pool control, the broader claim that AL is unnecessary under cheap full labeling is only partially supported.

Circularity Check

0 steps flagged

No circularity: empirical cost–performance comparison under a fixed, shared evaluation protocol; no derivation reduces to its inputs by construction.

full rationale

The paper’s load-bearing claims are experimental outcomes (FULL-LLM-26K under the decomposed interface beats FULL-HUMAN on macro-F1 / F1-Anti across four encoders and 10 seeds at ~1/10 cost; holistic ablation only ties; AL does not reliably beat random in the prefiltered pool; soft BB/DS evaluation preserves the ordering). The two-question rule is an operational definition applied identically to human and LLM annotators, not a redefinition of the evaluation metric. Training hyperparameters, acquisition functions, and cost accounting are shared or standard and do not force the reported ordering. Evaluation against majority-vote / soft human gold is a stated methodological choice with explicit limitations (§4.5–4.6, Limitations), not a self-definitional loop. There are no uniqueness theorems, fitted-then-predicted quantities, or load-bearing self-citation chains that reduce the central result to its inputs. The work is self-contained empirical comparison on a new dataset.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim is empirical and rests on standard supervised-learning and AL machinery plus an operational definition of the anti-immigrant label. Free parameters are ordinary training/AL hyperparameters shared across conditions; they do not selectively manufacture the LLM-vs-human gap. Domain assumptions concern the binary hostility definition, the adequacy of crowd majority vote as gold, and the representativeness of the Llama-prefiltered pool. No new physical or mathematical entities are postulated.

free parameters (3)
  • focal_loss_gamma = 2.0
    γ=2.0 chosen for class imbalance; standard but not derived from the data distribution of this corpus.
  • AL_seed_size_and_batch = 30 seed, k=50, 10 rounds
    30 seed + 50 per round × 10 rounds = 530; design choice following Fairstein et al., not fitted to maximize the claimed gap.
  • learning_rate_and_epochs = 2e-5
    AdamW 2e-5, batch 16, up to 10 epochs with early stopping; shared recipe across all conditions.
axioms (4)
  • domain assumption Anti-immigrant hostility is operationalized as the conjunction Q1=YES and Q2=YES under the two-question interface, applied identically to humans and LLMs.
    §3.2; the binary label is defined by this rule rather than a continuous or multi-class scheme.
  • domain assumption Majority-vote (and Bayesian soft) aggregation of six Prolific crowdworkers is a valid gold standard for evaluating classifier quality on this subjective task.
    §3.4, §3.7, Limitations; moderate α≈0.43–0.49 and demographic narrowness are acknowledged but the gold remains the evaluation target.
  • domain assumption The Llama-3.3-70B topic prefilter retains essentially all relevant anti-immigrant content (validated on 500 excluded items).
    §3.3 prefilter validation; AL-vs-random conclusions are conditioned on this enriched pool.
  • standard math Standard pool-based AL acquisition functions (Entropy, BALD, Core-Set, BADGE) and encoder fine-tuning with focal loss are appropriate for the comparison.
    §3.6–3.7; implementations follow cited literature.

pith-pipeline@v1.1.0-grok45 · 35295 in / 3260 out tokens · 31005 ms · 2026-07-12T20:33:19.855410+00:00 · methodology

0 comments
read the original abstract

Instruction-tuned LLMs can annotate thousands of instances at low cost. This raises two questions for active learning (AL): can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be cheaply labeled? We investigate both on a new dataset of 277,902 German political TikTok comments (25,974 LLM-labeled, 5,000 human-annotated), comparing LLM and human annotation across seven conditions, four encoders, and 10 random seeds. Under a two-question interface that mirrors the human annotation task, LLM annotation at scale outperforms human-supervised classifiers at roughly one-tenth the cost (\$28 for GPT-5.2 Batch API vs. \$316 for Prolific). The advantage holds for both a closed-source (GPT-5.2) and an open-weight (Qwen3.5-122B-10B) LLM, is robust under soft-label evaluation, and is unlocked specifically by the two-question decomposition; a holistic single-prompt baseline only ties with human supervision. AL provides no reliable advantage over random sampling under either LLM annotator. However, error structure varies sharply: only GPT-5.2 under the two-question interface produces classifiers with near-human FP/FN balance, while other LLM variants over-flag border-control and economic competition discourse. We release the dataset and code.

Figures

Figures reproduced from arXiv: 2604.13899 by Ahmad Dawar Hakimi, Hinrich Sch\"utze, Isabelle Augenstein, Lea Hirlimann.

Figure 1
Figure 1. Figure 1: Experimental framework. Human and LLM (Llama-3.3-70B prefiltering + GPT-5.2 classification) annotations feed into the same active learning loop, eval￾uated across seven conditions, four German encoder models, and 10 seeds on a party-stratified test set. longer AL versus human annotation but AL versus LLM annotation at scale. We test whether AL’s cost premium survives that comparison. We investigate this qu… view at source ↗
Figure 2
Figure 2. Figure 2: Learning curves for german_bert (top) and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Annotation cost–performance frontier for ger [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Left: FP/FN counts per condition for german_bert (mean ± 1 SD across 10 seeds; dots = individual seeds). Right: Errors ranked by prediction confidence for FULL-HUMAN and FULL-LLM-26K. Confidence on errors [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: LLM prompts. A.1: Prefiltering prompt ap￾plied to all 277,902 comments (temperature = 0, JSON output). A.2 Stage 2: Anti-Immigrant Annotation Prompt Prompt A.2: GPT-5.2 Anti-Immigrant Annotation You are an expert in sentiment analysis and political discourse, specializing in German-language text. Your task is to classify TikTok comments posted under videos from German political parties. Classify the commen… view at source ↗
Figure 6
Figure 6. Figure 6: A.2: Classification prompt applied to 25,974 prefiltered comments via GPT-5.2 Batch API (tempera￾ture = 0, structured JSON output). B LLM Infrastructure Details Llama-3.3-70B-Instruct (Stage 1). The model was run locally on four NVIDIA A100 80GB GPUs using vLLM (Kwon, 2025) with tensor par￾allelism across all four devices, bfloat16 preci￾sion, and a maximum sequence length of 512 tokens. Filtering all 277,… view at source ↗
Figure 7
Figure 7. Figure 7: t-SNE projection of all 25,974 immigration-relevant comments, clustered into 22 named discourse [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Learning curves for gbert-base (top) and xlm-r-base (bottom) on F1-Macro (left) and F1-Anti (right). [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: shows the predicted P(ANTI￾IMMIGRANT) distributions split by error type for german_bert. FP distributions for LLM-trained conditions peak substantially higher than for human-trained conditions, confirming that LLM-trained classifiers assign high confidence to their false positives. The pattern is consistent across all four encoder models. Correct FP FN 0.0 0.2 0.4 0.6 0.8 1.0 P(ANTI-IMMIGRANT) Full-Human C… view at source ↗
Figure 10
Figure 10. Figure 10: FP–FN imbalance by topic and condition for [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 5 linked inside Pith

  1. [1]

    InThe 7th Workshop on Online Abuse and Harms (WOAH), pages 231–242, Toronto, Canada

    Robust hate speech detection in social me- dia: A cross-dataset empirical evaluation. InThe 7th Workshop on Online Abuse and Harms (WOAH), pages 231–242, Toronto, Canada. Association for Computational Linguistics. David Arthur and Sergei Vassilvitskii. 2007. k- means++: the advantages of careful seeding. InPro- ceedings of the Eighteenth Annual ACM-SIAM S...

    arXiv 2007

  2. [2]

    InProceedings of the 13th International Workshop on Semantic Evaluation, pages 54–63, Min- neapolis, Minnesota, USA

    SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. InProceedings of the 13th International Workshop on Semantic Evaluation, pages 54–63, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics. Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prabhakaran. 2022. Dealing with disagreem...

    Pith/arXiv arXiv 2019

  3. [3]

    InProceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 77–86

    Class balancing for efficient active learning in imbalanced datasets. InProceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 77–86. Jan Fillies, Esther Theisen, Michael Hoffmann, Robert Jung, Elena Jung, Nele Fischer, and Adrian Paschke

  4. [4]

    Paula Fortuna and Sérgio Nunes

    A novel german tiktok hate speech dataset: far-right comments against politicians, women, and others.Discover Data, 3(1):4. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text.ACM Comput. Surv., 51(4):85:1–85:30. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertai...

    2018

  5. [5]

    Maarten Grootendorst

    Chatgpt outperforms crowd-workers for text- annotation tasks.CoRR, abs/2303.15056. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based TF-IDF procedure. CoRR, abs/2203.05794. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. Annollm: Making large lan...

    Pith/arXiv arXiv 2022

  6. [6]

    InMachine Learning and Knowledge Discovery in Databases

    Llms in the loop: Leveraging large language model annotations for active learning in low-resource languages. InMachine Learning and Knowledge Discovery in Databases. Applied Data Science Track - European Conference, ECML PKDD 2024, Vilnius, Lithuania, September 9-13, 2024, Proceedings, Part X, Lecture Notes in Computer Science, pages 397–

    2024

  7. [7]

    Aida Kostikova, Dominik Beese, Benjamin Paassen, Ole Pütz, Gregor Wiedemann, and Steffen Eger

    Springer. Aida Kostikova, Dominik Beese, Benjamin Paassen, Ole Pütz, Gregor Wiedemann, and Steffen Eger. 2024. Fine-grained detection of solidarity for women and migrants in 155 years of german parliamentary de- bates. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024...

    2024

  8. [8]

    Sarah Masud, Sahajpreet Singh, Viktor Hangya, Alexan- der Fraser, and Tanmoy Chakraborty

    Association for Computational Linguistics. Sarah Masud, Sahajpreet Singh, Viktor Hangya, Alexan- der Fraser, and Tanmoy Chakraborty. 2024. Hate per- sonified: Investigating the role of LLMs in content moderation. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pages 15847–15863, Miami, Florida, USA. As- sociatio...

    arXiv 2024

  9. [9]

    Paloma Piot, Patricia Martín-Rodilla, and Javier Parapar

    Automated annotation with generative AI re- quires validation.CoRR, abs/2306.00176. Paloma Piot, Patricia Martín-Rodilla, and Javier Parapar

    Pith/arXiv arXiv

  10. [10]

    InProceedings of the Eigh- teenth International AAAI Conference on Web and Social Media, ICWSM 2024, Buffalo, New York, USA, June 3-6, 2024, pages 2025–2039

    Metahate: A dataset for unifying efforts on hate speech detection. InProceedings of the Eigh- teenth International AAAI Conference on Web and Social Media, ICWSM 2024, Buffalo, New York, USA, June 3-6, 2024, pages 2025–2039. AAAI Press. Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and eval- uation. InProce...

    2024

  11. [11]

    CoRR, abs/1701.08118

    Measuring the reliability of hate speech an- notations: The case of the european refugee crisis. CoRR, abs/1701.08118. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet B. Pierrehumbert. 2022. Two contrasting data annota- tion paradigms for subjective NLP tasks. InProceed- ings of the 2022 Conference of the North American Chapter of the Association for Co...

    Pith/arXiv arXiv 2022

  12. [12]

    InProceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 2187– 2209, Dubrovnik, Croatia

    Investigating multi-source active learning for natural language inference. InProceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 2187– 2209, Dubrovnik, Croatia. Association for Computa- tional Linguistics. Miriah Steiger, Timir J. Bharucha, Sukrit Venkatagiri, Martin J. Riedl, and Matthew Le...

    Pith/arXiv arXiv 2021

  13. [13]

    Is this comment anti-immigrant:YES/NO?

    in the 25,974-comment immigration-relevant subset retained after Llama prefiltering. The 95 th percentile in the prefiltered subset motivates the 128-token truncation used in §3.7. Step Removed Duplicate IDs 20,390 Emoji-only 86,883 Empty 1,989 Mention-only 1,273 URL-only 447 Hashtag-only 406 Number-only 783 Laughter-only 270 Duplicate text (within-party)...

    2023

  14. [14]

    wann kommen die Grenzkontrollen?

    wir vergessen nicht” (‘Merkel 2015. . . we don’t forget’). The remainder (6/18) demand bor- der controls without referencing any group: “wann kommen die Grenzkontrollen?” (‘When are the bor- der controls coming?’), “Diese Grenze war vorher auch geschützt” (‘This border was protected before too’). In all 18 cases, the negativity targets political actors or...

    2015