Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A classifier trained purely on LLM-generated labels for media bias matches one trained on human expert labels, while cutting annotation cost.

desk verdict A useful, honest empirical study with release of 48k synthetic bias labels; the headline BABE result is partly confounded by the BABE-trained pre-filter, but the dataset and analysis deserve review. read the letter →

arxiv 2411.11081 v2 pith:DCJABTA4 submitted 2024-11-17 cs.CL

Tomas Horych , Christoph Mandl , Terry Ruas , Andre Greiner-Petter , Bela Gipp , Akiko Aizawa , Timo Spinde This is my paper · ORCID
classification cs.CL
keywords mediabiasdetectionLLMannotationsyntheticlabelsdatasetlabelingAnno-lexicalnewsclassificationMatthewscorrelationcoefficienttext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether large language models can replace human annotators for the subtle task of lexical media bias detection, and whether a smaller classifier can be trained on their output. The authors build Anno-lexical, a dataset of 48,330 news sentences labeled by majority vote of three open-source LLMs prompted with eight human-labeled examples each. A RoBERTa classifier fine-tuned on those synthetic labels reaches a Matthews correlation coefficient (MCC) of 0.662 on the BABE benchmark, close to the 0.678 of a classifier trained on human labels, and beats the human-trained model on the out-of-distribution BASIL set (0.205 vs 0.174) while surpassing each annotating LLM by 5 to 9 points of MCC. The point of the exercise is cost: if synthetic annotations are good enough, large training sets for bias detection can be built for a fraction of expert-labeling expense. The paper also documents the trade-off: the synthetic-trained model has lower precision and worse robustness to input perturbations, and its advantage shrinks when training sets are matched in size.

What carries the argument

The central mechanism is the Anno-lexical pipeline: select three LLM annotators by their MCC on a human-labeled development set; annotate a politically balanced corpus of news sentences with few-shot in-context prompts that include up to eight human-labeled examples retrieved by the KATE similarity algorithm, each accompanied by a GPT-4-written explanation; aggregate the three per-sentence labels by majority vote; then fine-tune a RoBERTa encoder with a two-layer classification head on the resulting 48,330 labeled sentences. The same architecture is used for the human-annotation fine-tuned baseline so that the only difference is the source of the labels. Before annotation, a pre-trained bias classifier is used to downsample each political segment to 50% estimated-biased and 50% estimated-neutral sentences.

What would settle it

Re-annotate a random sample of Anno-lexical with expert human annotators and compare agreement with the LLM majority-vote labels separately for each political-leaning segment; if agreement is no better than chance on any segment, the claim that synthetic annotations carry the same signal as human labels fails. Alternatively, rebuild the corpus using a different bias estimator for the pre-annotation downsampling: if SA-FT's BABE/BASIL results change materially, the outcome depends on the estimator's specific errors rather than on the LLMs' annotation ability.

Watch

Extended reading notes

Core claim

The paper's central claim is that a downstream classifier fine-tuned on synthetic annotations from an ensemble of LLMs (SA-FT) performs comparably to a conventional classifier fine-tuned on human annotations (HA-FT) for sentence-level lexical bias detection. On the BABE test set, SA-FT reaches an MCC of 0.662 versus 0.678 for HA-FT; on the held-out BASIL set, SA-FT reaches 0.205 versus 0.174, a difference the authors report as statistically significant by a paired significance test at p<0.05. SA-FT also beats each of its three teacher LLMs by 5 to 9 points of MCC despite having roughly 130 million parameters versus the teachers' 7 to 13 billion. The paper presents this as evidence that LLM annotations can substitute for a large portion of human labeling effort in media-bias dataset construction, while noting that SA-FT has lower precision and worse invariance robustness, and that its advantage over HA-FT disappears when training data are matched in size.

Load-bearing premise

The whole experiment assumes that the pre-trained bias classifier used to balance the corpus before annotation gives correct enough estimates that a 50/50 split of 'estimated biased' and 'estimated neutral' sentences actually balances the label distribution; if that estimator is wrong in a way that correlates with political leaning, the synthetic dataset and the downstream model inherit its errors instead of measuring LLM annotation quality.

Editorial extensions

If this is right

  • On the two established benchmarks, a classifier trained only on majority-voted LLM annotations is competitive with one trained on expert labels: 0.662 vs 0.678 MCC on BABE and 0.205 vs 0.174 on BASIL.
  • A small downstream model can distill the ensemble: SA-FT improves on the annotators' majority vote by 2.3% MCC and on each individual LLM by 5 to 9 points, while needing far fewer parameters at inference.
  • Scale is part of the recipe: when Anno-lexical is cut to the same 3k size as the human-labeled training set, SA-FT drops to 0.638 MCC, about 4% below HA-FT, so the full 34k training split is what closes the gap.
  • The dataset release, 48,330 balanced sentences with synthetic labels, is itself a resource for further media-bias classification work, with the caveat that robustness to entity substitutions is weaker than for human-trained models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the main driver of SA-FT's success is dataset size rather than label source, the same synthetic-annotation recipe should transfer to other subjective classification tasks whose bottleneck is expert annotation cost.
  • Beyond the paper: the pre-annotation balancing makes the final dataset conditional on the error profile of the bias estimator used; a different estimator could produce a different Anno-lexical and different downstream rankings, a dependency the paper does not test.
  • Beyond the paper: mixing a small amount of human labels into the synthetic training set would likely raise precision and invariance robustness, since those are precisely the dimensions where SA-FT falls short.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a pipeline for creating large-scale training datasets for sentence-level lexical media bias detection using LLM-generated synthetic annotations. The authors select three open-source LLMs after benchmarking on BABE, annotate a corpus of 64,712 sentences (later reduced to 48,330) via majority vote, and fine-tune a RoBERTa classifier on the resulting Anno-lexical dataset. They compare this SA-FT model against a conventionally fine-tuned HA-FT model on the BABE and BASIL benchmarks and via a CheckList-style behavioral stress test. The paper reports that SA-FT outperforms all individual annotator LLMs and achieves MCC 0.662 on BABE versus 0.678 for HA-FT, while surpassing HA-FT on BASIL (0.205 versus 0.174) and showing trade-offs in precision and robustness.

Significance. If the headline result held, the paper would make a useful practical contribution by showing that LLM-generated annotations can substitute for expert labels in media-bias dataset construction, with substantially lower cost. The release of the Anno-lexical dataset, the Annomatic annotation tool, and the code is a concrete asset for reproducibility, and the behavioral stress testing is a valuable addition compared with typical benchmark-only evaluations. However, the central comparison is weakened by several load-bearing confounds: the downstream BABE evaluation is entangled with the use of BABE-trained MAGPIE for corpus pre-selection, the SA-FT versus HA-FT comparison is not size-matched, and the entire empirical evaluation rests on a single run without uncertainty estimates. The paper's own coreset experiment shows that equalizing training size flips the BABE comparison in favor of HA-FT, so the current framing overstates the evidence that synthetic annotations are of comparable quality to human labels. The contribution is better described as a promising pilot with important caveats than as a definitive demonstration of parity.

major comments (4)
  1. [Section 5.1 and Table 2] The BABE test comparison is confounded by the corpus construction. In Section 5.1, the authors use MAGPIE—a media bias classifier trained on BABE—to pre-classify roughly 400,000 candidate sentences and then downsample so that each political segment contains exactly 50% estimated-biased sentences. This means the resulting Anno-lexical training distribution is shaped by a model trained on the very benchmark whose test set is later used to claim that SA-FT performs comparably to HA-FT. The few-shot examples and annotator selection also come from BABE (Sections 4.1 and 4.2). As reported, the BABE MCC comparison (0.662 versus 0.678) does not isolate the quality of LLM annotations; it evaluates a pipeline whose training data distribution is partially determined by the test benchmark's training split. The paper should either report agreement/disagreement between MAGPIE's pre-classification and the LLM majority labels on the pre-balancing pool, or run an ablation that omits the MAGPIE-based pre-selection, or restrict the parity claim to the BASIL evaluation.
  2. [Section 6.3 and Table 2] The headline SA-FT versus HA-FT comparison is not size-matched and the paper's own coreset result undermines the 'comparable quality' claim. SA-FT is trained on 34k synthetic annotations while HA-FT is trained on 3k human annotations. When the authors train SA-FT on a 3k coreset (SA-FT coreset), it achieves MCC 0.638 on BABE, 4 points below HA-FT's 0.678. This indicates that the apparent parity on BABE is largely driven by the greater size of the synthetic dataset, not by annotation quality. The claim in the abstract and Section 6.3 that SA-FT 'performs close to' HA-FT should be reframed to acknowledge that, at matched training size, the synthetic pipeline is inferior on BABE and only the larger data volume compensates.
  3. [Section 6.2] All experimental results are based on a single run, as stated in Section 6.2: 'All training and evaluations were run as a single run.' The reported differences between SA-FT and HA-FT on BABE (0.662 versus 0.678) and on BASIL (0.205 versus 0.174) are therefore presented without any uncertainty estimate, confidence interval, or sensitivity to random seeds. The McNemar test reported in Section 6.3 is applied only to the BASIL comparison and cannot account for variance in model training. The authors should either report multiple random seeds with means and standard deviations or provide a clear justification for why single-run results are sufficient for the claimed parity.
  4. [Section 6.3 and Table 2] The BASIL result is presented as a significant advantage, but the absolute performance is near chance and the operating point is highly asymmetric. On BASIL, SA-FT obtains MCC 0.205 with precision 0.171 and recall 0.502, while HA-FT obtains MCC 0.174 with precision 0.169 and recall 0.384. The paper itself notes that both models partially confuse informational bias with lexical bias. A 3.1-point MCC difference between two near-chance classifiers, measured on a single run, is weak evidence for the superiority of the synthetic pipeline. The discussion should more prominently state that the BASIL comparison is exploratory, and the claim that SA-FT 'outperforms' HA-FT should be tempered accordingly.
minor comments (6)
  1. [Abstract, Section 6, Table 2] The metric is consistently misspelled as 'Mathew's Correlation Coefficient' in the Abstract and elsewhere; it should be 'Matthews Correlation Coefficient'.
  2. [Figure 1] The figure caption contains a typo: 'Human-Annotation Ftine-Tuning' should read 'Human-Annotation Fine-Tuning'.
  3. [Section 6.3] The sentence 'the HA-FT model, still outperforms the synthetic model by 1,5%' uses a decimal comma; for consistency with the rest of the paper, this should be '1.5%'.
  4. [Section 4.2 and Table 1] Model names are used inconsistently (e.g., 'LLama 2 13B Chat' vs. 'Llama 2 13B Chat' in Table 4). Please standardize the capitalization and naming of all models across the text, tables, and appendix.
  5. [Section 5.1 and Figure 3] The figure caption and surrounding text refer to 'party/label distribution' and 'political ideology distribution' interchangeably; 'party' is not defined and may be misleading. Consider using 'political leaning segment' consistently.
  6. [Section 6.3] The phrase 'reducing the cost of deployment by a factor of 1005 or 300' is unclear because the comparison basis (parameter count, inference cost, or energy) is not specified. Please clarify the calculation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BABE-derived components create a distributional confound but no step reduces to its own input by construction.

full rationale

The paper's central claim is that a classifier fine-tuned on LLM-generated annotations (SA-FT) approaches or exceeds a classifier fine-tuned on human labels (HA-FT). No step in the derivation chain equates a prediction to a fitted input. The annotator LLMs are ranked on the BABE training split and the few-shot prompt examples are drawn from BABE training data, but the BABE test labels are never used to construct Anno-lexical. The pre-classification stage uses MAGPIE (Horych et al., 2024) to balance the corpus by estimated bias; MAGPIE is an independently trained and publicly released model, and its outputs are only a sampling prior, not the synthetic labels that train SA-FT. The final training labels come from the LLM majority vote, and the evaluation on BABE and BASIL compares models on held-out benchmark instances. The BASIL result and the CheckList stress tests provide additional external checks. The self-citation to MAGPIE is not a load-bearing circular justification because the paper's contribution does not depend on MAGPIE's outputs being correct in a way that presupposes the paper's conclusion. The BABE-derived choices do create a distributional confound and weaken the claim that BABE is a fully external validation, but that is a correctness or validity concern, not a circularity of derivation. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central result depends on several assumptions about label quality and corpus construction. The free parameters are the prompt design choices (k=8, annotator set) tuned on the BABE dev set, and the enforced label balance. The key axioms are that BABE and BASIL provide a valid operationalization of lexical bias, that GPT-4 explanations are accurate, that the three-LLM majority vote is a good proxy for human labels, and that MAGPIE's pre-classification is accurate enough for corpus balancing. No invented entities are introduced.

free parameters (4)
  • k (number of few-shot examples) = 8
    Selected because the 8-shot explanation prompting setting achieved the highest mean MCC on the BABE train/dev set (Table 1, Section 4.2).
  • Annotator ensemble composition = Zephyr 7B beta, OpenChat 3.5, Llama 2 13B Chat
    Chosen greedily as the top-performing open-source models on the BABE train/dev set while excluding costly models (Section 4.2).
  • Bias label balance ratio = 50% biased / 50% unbiased per political segment
    Enforced via MAGPIE pre-classification and post-annotation downsampling (Sections 5.1 and 5.2); this choice shapes the dataset distribution.
  • Corpus size after downsampling = 48,330 sentences
    Result of balancing across five political segments and bias labels, with 64,712 sentences annotated before the final 1:1 downsample (Section 5.2).
assumptions (5)
  • domain assumption BABE labels are a reliable operationalization of lexical bias.
    BABE is used as the human-labeled ground truth for training HA-FT, for few-shot examples, and for annotator selection (Sections 2 and 4.2). The paper does not question BABE's definition of bias.
  • domain assumption GPT-4 generated explanations for the in-context examples are accurate.
    Explanations are part of the prompt (Table 5) and guide the LLM annotators. No validation of explanation quality is reported (Section 4.1).
  • domain assumption The majority vote of three open-source LLMs produces valid labels for unseen news sentences.
    The paper assumes the ensemble's consensus is a sufficient proxy for human labels on Anno-lexical (Section 4.1).
  • ad hoc to paper MAGPIE's pre-classification accurately estimates lexical bias for corpus balancing.
    MAGPIE, trained on BABE, is used to select which sentences enter Anno-lexical (Section 5.1). Its errors influence the dataset composition and downstream results.
  • domain assumption Mapping BASIL's informational bias label to the negative class is acceptable for comparison.
    BASIL's ternary labels are collapsed to binary, treating informational bias as not lexical bias (Section 6.1). This mapping affects the reported MCC.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection." pith.science (2026). https://pith.science/paper/DCJABTA4

@misc{pith2026241111081,
  author       = {Pith},
  title        = {Pith review of: The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DCJABTA4}},
  note         = {Machine review of arXiv:2411.11081}
}
read the original abstract

High annotation costs from hiring or crowdsourcing complicate the creation of large, high-quality datasets needed for training reliable text classifiers. Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. LLMs have shown promising results in annotating downstream tasks like hate speech detection and political framing. Building on the success in these areas, this study investigates whether LLMs are viable for annotating the complex task of media bias detection and whether a downstream media bias classifier can be trained on such data. We create annolexical, the first large-scale dataset for media bias classification with over 48000 synthetically annotated examples. Our classifier, fine-tuned on this dataset, surpasses all of the annotator LLMs by 5-9 percent in Matthews Correlation Coefficient (MCC) and performs close to or outperforms the model trained on human-labeled data when evaluated on two media bias benchmark datasets (BABE and BASIL). This study demonstrates how our approach significantly reduces the cost of dataset creation in the media bias domain and, by extension, the development of classifiers, while our subsequent behavioral stress-testing reveals some of its current limitations and trade-offs.

Figures

Figures reproduced from arXiv: 2411.11081 by the authors.

Figure 1
Figure 1. Workflow diagram presenting the difference between the two approaches to fine-tuning the model - [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The workflow diagram describing an end-to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The figure demonstrates the transformation of the unlabeled corpus (left) to the final [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A hybrid system using a calibrated SVM plus selective GPT-4o fallback detects prosocial game chat at roughly 0.90 precision while cutting LLM inference cost by about 70%.

  2. Are Large Language Models the future crowd workers of Linguistics?

    cs.CL 2025-02 conditional novelty 5.0 of 10

    In two replicated linguistics experiments, GPT-4o-mini's zero-shot responses matched or beat published human performance, but the study lacks statistical validation and relies on only two tasks.

Reference graph

Works this paper leans on

51 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Meysam Alizadeh, Ma \"e l Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2023. https://doi.org/10.48550/arXiv.2307.02179 Open- Source Large Language Models Outperform Crowd Workers and Approach ChatGPT in Text-Annotation Tasks . Computation and Language, (arXiv:2307.02179)

    arXiv doi:10.48550/arxiv.2307.02179 2023

  4. [4]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B : an open large language model with state-of-the-art performance

    2023

  5. [5]

    Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2021. https://doi.org/10.18653/v1/2021.eacl-main.29 A systematic review of reproducibility research in natural language processing . In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 381--393, Online. Associati...

    doi:10.18653/v1/2021.eacl-main.29 2021

  6. [6]

    Anya Belz, Craig Thomson, Ehud Reiter, Gavin Abercrombie, Jose M. Alonso-Moral, Mohammad Arvan, Anouck Braggaar, Mark Cieliebak, Elizabeth Clark, Kees van Deemter, Tanvi Dinkar, Ond r ej Du s ek, Steffen Eger, Qixiang Fang, Mingqi Gao, Albert Gatt, Dimitra Gkatzia, Javier Gonz \'a lez-Corbelle, Dirk Hovy, Manuela H \"u rlimann, Takumi Ito, John D. Kellehe...

    2023

  7. [7]

    Chengliang Chai, Jiayi Wang, Nan Tang, Ye Yuan, Jiabin Liu, Yuhao Deng, and Guoren Wang. 2023. https://doi.org/10.1145/3580305.3599326 Efficient Coreset Selection with Cluster-based Methods . In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, pages 167--178, New York, NY, USA. Association for Computing Machinery

    arXiv 2023

  8. [8]

    Davide Chicco and Giuseppe Jurman. 2020. https://doi.org/10.1186/s12864-019-6413-7 The advantages of the Matthews correlation coefficient ( MCC ) over F1 score and accuracy in binary classification evaluation . BMC Genomics, 21(1):6

    doi:10.1186/s12864-019-6413-7 2020

Show all 51 references
  1. [10]

    Michael Chmielewski and Sarah C. Kucker. 2020 b . https://doi.org/10.1177/1948550619875149 An MTurk Crisis ? Shifts in Data Quality and the Impact on Study Results . Social Psychological and Personality Science, 11(4):464--473

    2020 doi

  2. [11]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent...

    2022 doi arXiv

  3. [12]

    Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. https://doi.org/10.18653/v1/D19-1664 In Plain Sight : Media Bias Through the Lens of Factual Reporting . In Proceedings of the 2019 Conference on Empirical Methods in Natu...

    2019 doi

  4. [13]

    Fabrizio Gilardi, Meysam Alizadeh, and Ma \"e l Kubli. 2023. https://doi.org/10.1073/pnas.2305016120 ChatGPT outperforms crowd workers for text-annotation tasks . Proceedings of the National Academy of Sciences, 120(30):e2305016120

    2023 doi

  5. [14]

    Laurence Gillick and Stephen J Cox. 1989. Some statistical issues in the comparison of speech recognition algorithms. In International Conference on Acoustics, Speech, and Signal Processing,, pages 532--535. IEEE

    1989

  6. [15]

    Shahriar Golchin and Mihai Surdeanu. 2023. https://doi.org/10.48550/ARXIV.2308.08493 Time travel in llms: Tracing data contamination in large language models . CoRR, abs/2308.08493

    2023 doi arXiv

  7. [16]

    Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. https://doi.org/10.18653/v1/2024.naacl-industry.15 AnnoLLM : Making Large Language Models to Be Better Crowdsourced Annotators . In Proceedings of ...

    2024 doi

  8. [17]

    Tomas Horych. 2022. Metody detekce vyvazenosti zpravodajskychch textu

    2022

  9. [18]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation

    Tomas Horych, Martin Wessel, Jan Philip Wahle, Terry Ruas, Jerome Wassmuth, Andre Greiner-Petter, Akiko Aizawa, Bela Gipp, and Timo Spinde. 2024. https://aclanthology.org/2024.lrec-main.952 Magpie: Multi-task analysis of media-bias generalization with pre-trained identificatio...

    2024

  10. [19]

    Fan Huang, Haewoon Kwak, and Jisun An. 2023. https://doi.org/10.1145/3543873.3587368 Is ChatGPT better than Human Annotators ? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech . In Companion Proceedings of the ACM Web Conference 2023 , pages 294--297, Au...

    2023

  11. [20]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  12. [21]

    Jan-Christoph Klie, Richard Eckart de Castilho , and Iryna Gurevych. 2023. https://doi.org/10.48550/arXiv.2307.08153 Analyzing Dataset Annotation Quality Management in the Wild

    2023 doi arXiv

  13. [22]

    David Krieger, Timo Spinde, Terry Ruas, Juhi Kulshrestha, and Bela Gipp. 2022. https://doi.org/10.1145/3529372.3530932 A Domain -adaptive Pre -training Approach for Language Bias Detection in News . In 2022 ACM / IEEE Joint Conference on Digital Libraries ( JCDL ) , Cologne, Germany

    2022

  14. [23]

    Yuanyuan Lei and Ruihong Huang. 2024. https://doi.org/10.48550/arXiv.2404.01722 Sentence-level Media Bias Analysis with Event Relation Graph

    2024 doi arXiv

  15. [24]

    Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565--6576. PMLR

    2021

  16. [25]

    Luyang Lin, Lingzhi Wang, Xiaoyan Zhao, Jing Li, and Kam-Fai Wong. 2024. IndiVec : An Exploration of Leveraging Large Language Models for Media Bias Detection with Fine-Grained Bias Indicators . In Findings of the Association for Computational Linguistics : EACL 2024 , pages 1...

    2024

  17. [26]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 What Makes Good In-Context Examples for GPT-3 ? In Proceedings of Deep Learning Inside Out ( DeeLIO 2022): The 3rd Workshop on Knowledge Ext...

    2022 doi

  18. [27]

    Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi-Yu . 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.334 Active Learning Principles for In-Context Learning with Large Language Models . In Findings of the Association for Computational Linguistics : EMN...

    2023 doi

  19. [28]

    Marshall, Partha S.R

    Catherine C. Marshall, Partha S.R. Goguladinne, Mudit Maheshwari, Apoorva Sathe, and Frank M. Shipman. 2023. https://doi.org/10.1145/3578503.3583622 Who Broke Amazon Mechanical Turk ? An Analysis of Crowdsourcing Data Quality over Time . In Proceedings of the 15th ACM Web Scie...

    2023

  20. [29]

    MistralAI. 2024. https://mistral.ai/news/mixtral-of-experts/ Mixtral of experts: Dynamic gating for efficient model scaling

    2024

  21. [30]

    Robert Munro Monarch. 2021. Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster

    2021

  22. [31]

    Roberto Navigli, Simone Conia, and Bj \"o rn Ross. 2023. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality, 15(2):1--21

    2023

  23. [32]

    OpenAI. 2023. https://platform.openai.com/docs/models/gpt-3-5-turbo Gpt-3.5

    2023

  24. [33]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  25. [34]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  26. [35]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118

    2020 arXiv

  27. [36]

    Francisco-Javier Rodrigo-Ginés, Jorge Carrillo de Albornoz, and Laura Plaza. 2024. https://doi.org/https://doi.org/10.1016/j.eswa.2023.121641 A systematic review on media bias detection: What is media bias, how it is expressed, and how to detect it . Expert Systems with Applic...

    2024

  28. [37]

    Morgan Klaus Scheuerman, Alex Hanna, and Emily Denton. 2021. https://doi.org/10.1145/3476058 Do datasets have politics? disciplinary values in computer vision dataset development . Proc. ACM Hum.-Comput. Interact., 5(CSCW2)

    2021 doi

  29. [38]

    Timo Spinde, Smilla Hinterreiter, Fabian Haak, Terry Ruas, Helge Giese, Norman Meuschke, and Bela Gipp. 2023. The media bias taxonomy: A systematic literature review on the forms and automated detection of media bias. arXiv preprint arXiv:2312.16148

    2023 arXiv

  30. [40]

    Timo Spinde, David Krieger, Manu Plank, and Bela Gipp. 2021 b . https://doi.org/10.1109/JCDL52503.2021.00053 Towards A Reliable Ground - Truth For Biased Language Detection . In Proceedings of the ACM / IEEE - CS Joint Conference on Digital Libraries ( JCDL ) , Virtual Event

    2021

  31. [41]

    Timo Spinde, Jan-David Krieger, Terry Ruas, Jelena Mitrović, Franz Götz-Hahn, Akiko Aizawa, and Bela Gipp. 2022. https://doi.org/10.1007/978-3-030-96957-8_20 Exploiting transformer-based multitask learning for the detection of media bias in news articles . In Proceedings of th...

    2022 doi

  32. [42]

    Timo Spinde, Manuel Plank, Jan-David Krieger, Terry Ruas, Bela Gipp, and Akiko Aizawa. 2021 c . https://doi.org/10.18653/v1/2021.findings-emnlp.101 Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts . In Findings of the Association fo...

    2021 doi

  33. [43]

    Timo Spinde, Lada Rudnitckaia, Sinha Kanishka, Felix Hamborg, Bela , Gipp , and Karsten Donnay. 2021 d . https://doi.org/10.6084/m9.figshare.17192924 MBIC – a media bias annotation dataset including annotator characteristics . In Proceedings of the iConference 2021 , Beijing, ...

    2021 doi

  34. [44]

    Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. http://arxiv.org/abs/2402.13446 Large language models for data annotation: A survey

    2024 arXiv

  35. [45]

    Petter T \"o rnberg. 2023. https://doi.org/10.48550/arXiv.2304.06588 ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

    2023 doi arXiv

  36. [46]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  37. [47]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. http://arxiv.org/abs/2310.16944 Zephy...

    2023 arXiv

  38. [48]

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235

    2023 arXiv

  39. [49]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  40. [50]

    Martin Wessel and Tom \'a s Horych. 2024. https://aclanthology.org/2024.ltedi-1.3 Beyond the surface: Spurious cues in automatic media bias detection . In Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion, pages 21--30, St. Julian's, ...

    2024

  41. [51]

    Martin Wessel, Tomás Horych, Terry Ruas, Akiko Aizawa, Bela Gipp, and Timo Spinde. 2023. https://doi.org/10.1145/3539618.3591882 Introducing MBIB - The First Media Bias Identification Benchmark Task and Dataset Collection . In Proceedings of the 46th International ACM SIGIR Co...

    2023

  42. [52]

    Endalkachew Yitayew. 2023. https://www.yitay.net/blog/flan-ul2-20b Flan-ul2 20b: Exploring google's latest language model

    2023

  43. [53]

    Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Computational Linguistics, 50(1):237--291

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.