Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

In Plain Sight: Media Bias Through the Lens of Factual Reporting

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Bias in news is mostly in selected facts, not loaded words

desk verdict A genuinely useful annotated resource for a real blind spot in bias detection, with a caveat that its headline prevalence claim rests on annotation discussions that start from very low agreement. read the letter →

arxiv 1909.02670 v1 pith:BF4GYLQ2 submitted 2019-09-05 cs.CL

classification cs.CL
keywords mediabiasinformationallexicalnewsframingannotationBASILdatasetpoliticalideologyspan
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that political bias in news is not primarily a matter of loaded words or syntax. Using a new dataset of 300 articles—100 identical events covered by Fox News, the New York Times, and Huffington Post—it finds that about 72% of biased passages are informational: factual or background details selected to sway the reader, rather than lexically biased phrasing. If true, automatic bias detection must model what content is included and how it frames an entity, not just word choice. The paper also documents where bias sits in articles and offers a BERT-based baseline showing informational bias is harder to detect than lexical bias.

What carries the argument

The load-bearing artifact is BASIL, a span-level annotation dataset built from 100 article triplets, each triplet reporting the same event in Fox News, the New York Times, and Huffington Post. Annotators, blind to media source, mark every biased span as either lexical (removable or rephrasable without changing meaning) or informational (removable only by changing content or context), with additional labels for target, polarity, aim, and whether the span appears in a quote. The triplet design is essential: comparing three reports of the same event lets annotators distinguish event-relevant facts from peripheral details that shade opinion, and lets the paper measure outlet-level patterns while holding the story constant. The same structure powers the main baseline, a BERT model fine-tuned per bias type.

What would settle it

Re-annotate the same 100 article triplets with two groups of annotators of opposite political leanings who do not discuss or see outlet names, then compare the percentage of informational versus lexical spans. If the 72/28 split changes substantially by group ideology—or if the same article yields systematically different counts of informational spans—the prevalence finding would be an artifact of the annotation process rather than a stable property of news writing.

Watch

Extended reading notes

Core claim

The central discovery is that informational bias—sentences or clauses conveying tangential, speculative, or background facts that shape opinion about a political entity—appears in news more frequently than lexical bias. Across 1,727 annotated bias spans in the BASIL dataset, 1,249 (72.3%) are informational and 478 (27.7%) are lexical, a pattern consistent across all three outlets. While lexical bias clusters in the opening quarter of articles, informational bias is spread through the whole text, and almost half of it arrives inside quotations, suggesting outlets use selected quotes as a proxy for their own stance. In entity-level analysis, negative informational coverage tracks the outlet's ideology: conservative entities receive more negative informational bias in NYT and HPO, and liberal entities receive more in FOX.

Load-bearing premise

The central claim depends on the post-discussion labels reflecting recoverable properties of the text rather than the annotators' shared political beliefs; raw pairwise agreement on the exact text spans was low (roughly 13 to 21 on a 0 to 100 F1 scale), and the paper defers study of how annotators' prior beliefs affect their judgments.

Editorial extensions

If this is right

  • Bias detection systems that only look at word choice will miss most bias; modeling content selection and article-level context is necessary.
  • Quotations are a major channel for informational bias, so the editorial choice of which quotes to include should be treated as a strategic bias signal.
  • Lexical bias tends to appear early in articles while informational bias is distributed throughout, so discourse position can help distinguish the two types.
  • Span-level annotations reveal outlet slant toward specific political figures, not just an overall ideological leaning, enabling more granular media analysis.
  • The sentence-level TF-IDF baseline's modest performance suggests that detecting informational bias requires information beyond the sentence being judged.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the prevalence finding holds, media literacy guidance should teach readers to ask what facts were included, omitted, or placed in quotes, not only to scan for emotionally charged words.
  • The triplet-comparison method itself could be turned into a detection strategy: flag factual details that appear in only one outlet's account of the same event as candidate informational bias.
  • Because raw pairwise annotator agreement on spans is low and the paper defers the study of annotators' prior beliefs, a replication with politically balanced annotator groups would test whether the 72/28 split is a stable property of news text or partly a product of the annotation process.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces BASIL, a dataset of 300 news articles organized as 100 triplets (Fox News, New York Times, Huffington Post) reporting the same event, annotated at the span level for two types of bias: lexical bias (bias from word choice and syntax) and informational bias (factual content selected or contextualized to sway opinion). The authors report that informational bias spans outnumber lexical bias spans in all three outlets (72.3% vs. 27.7% overall), analyze differences in positional distribution, quote usage, and entity portrayal, and benchmark BERT and rule-based baselines for sentence-level and token-level bias detection. The central claim is that informational bias is more prevalent than lexical bias and that this finding motivates a shift in bias-detection research from surface linguistic cues to content selection and context.

Significance. If the prevalence claim is sound, the paper makes a meaningful contribution by refocusing media-bias research on content-level framing rather than lexical markers, and the BASIL dataset itself is a valuable resource with fine-grained annotations (bias type, target, polarity, aim, quote status). The experiments are honestly reported with 10-fold cross-validation, variance ranges, and a publicly available annotation tool. The descriptive statistics and entity-portrayal analysis offer a rich view of outlet-specific bias. However, the load-bearing prevalence claim depends entirely on the validity of a gold standard produced by two annotators who resolve large initial disagreements through discussion; the paper's own IAA numbers show raw span agreement is very low, and the reported improvement to gold-standard agreement does not demonstrate external validity. This makes the main conclusion provisional and in need of independent validation.

major comments (2)
  1. [Section 3, Table 3] The central claim that informational bias outnumbers lexical bias is a tally of gold-standard labels, but the gold standard is generated by two annotators who first disagree substantially (exact-match span F1 for informational bias ranges from 12.68 to 20.68, and lenient lexical F1 is at most 15.38) and then resolve disagreements via discussion. The paper states that individual-to-gold F1 rises to 0.70 (informational) and 0.56 (lexical), but because the gold standard is produced by the same two annotators, this only demonstrates convergence to a shared interpretation, not that the shared interpretation tracks recoverable text properties. Footnote 2 explicitly postpones study of annotators' prior beliefs, leaving open the possibility that the discussion process systematically inflates the novel 'informational' category relative to 'lexical.' To support the prevalence claim, the authors should provide independent evidence: for example, a fresh-team re-annotation of a sample, an adjudication protocol showing that disagreements are resolved by reference to textual criteria external to the annotators, or an analysis of how disagreement resolution shifts the type distribution. Without such evidence, the headline 72.3% versus 27.7% split could be an artifact of the annotation procedure.
  2. [Section 5, footnote 5] The TF-IDF baseline selects 'the 4 sentences in each article with the lowest average TF-IDF token scores,' with the justification that 'BASIL averages 4.1 informational bias spans per article.' This calibrates the baseline's only free parameter (k=4) directly to the gold-standard mean, giving the baseline oracle information about the number of spans. The comparison between this baseline and BERT is therefore not a fair parameter-free baseline; the claim that the baseline 'performs relatively well' and 'indicat[es] the importance of explicitly modeling context' is weakened because k is chosen to fit the dataset. The authors should either report results for several k values (e.g., k=2, 4, 6, 8) or remove the dataset-dependent calibration and discuss the sensitivity of the conclusion.
minor comments (4)
  1. [Section 4.1, Figure 2] The claim that 'lexical bias appears early in an article' and 'informational bias ... can appear at any position' is based on visual inspection of Figure 2; a statistical test (e.g., a chi-square or permutation test on quartile distributions) would make this quantitative claim more rigorous.
  2. [Section 3, Table 3] In Table 3, the 'B + C' row for lexical bias has dashes for exact matching because no exact lexical spans were shared; this is fine, but the paper should explicitly state whether the lenient-row F1=15.38 for B+C is based on one or few matches, given the small number of resolved lexical spans for that pair.
  3. [Section 5, Table 2] The sentence reports that the standard deviation across BERT folds ranges from 3.36 to 12.44; it would be clearer to report the standard deviation for each model in Table 2 rather than a global minimum and maximum, since some of the central comparisons (e.g., informational vs. lexical sentence F1) may overlap within noise.
  4. [Abstract and Figure 1] The abbreviation 'HPO' is used in Figure 1 and throughout, but it is only expanded as 'Huffington Post' in the text; consider defining it at first use in the caption or text for readers of the standalone figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the prevalence claim is a direct annotation tally, and no load-bearing step reduces to its own inputs.

full rationale

The paper's central descriptive claim—that informational bias spans outnumber lexical spans (Table 1)—is a direct count over the BASIL gold annotations, not the output of a fitted model or a theorem derived from the annotation definitions. The definition of informational bias (sentences/clauses that are tangential, speculative, or background) and lexical bias (bias from word choice and syntax) does not logically fix the span-count ratio; annotators could in principle have found more lexical spans. The only place where a quantity is calibrated to the gold data is the TF-IDF baseline, which chooses k=4 sentences per article because 'BASIL averages 4.1 informational bias spans per article' (footnote 5). This is explicitly a simple baseline for the detection experiment, not a prediction supporting the prevalence claim, and its reported sentence-level F1 of 26.02 is presented as a low benchmark rather than as evidence for the prevalence result. No load-bearing argument relies on self-citation: the cited prior work is external (e.g., Budak et al., Card et al., Recasens et al.), and no uniqueness result or ansatz is imported from the present authors' own prior papers. The acknowledged limitation (footnote 2: 'The likely effect of annotators' prior beliefs on their perception of bias will be investigated in future work') and the low raw span-level IAA (Table 3) are threats to the external validity of the annotation-based findings, but they do not make the derivation circular. The BERT experiments are independent benchmarks whose scores are reported as task-difficulty evidence, not renamed as confirmations of the prevalence claim. Hence no circular step is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central analysis rests on the construct of informational bias, the validity of adjudicated annotations, and a convenience sample of 100 triplets. No free parameter is fitted to make the main prevalence claim true; the only internal calibration is the k=4 TF-IDF baseline. The paper therefore contributes data and descriptive analysis rather than a derivation, and its assumptions are mostly empirical and definitional.

free parameters (1)
  • k, number of low-TF-IDF sentences selected by the baseline = 4
    Section 5, sentence-level classifier: the baseline selects 4 sentences per article because BASIL averages 4.1 informational bias spans per article. This ties the baseline's expected recall to the dataset's annotation rate rather than to an independent model.
assumptions (4)
  • domain assumption Informational bias, defined as 'sentences or clauses that convey information tangential, speculative, or as background to the main event in order to sway readers' opinions', is a real and identifiable property of text.
    This is the central construct of the paper; its existence is operationalized only through the authors' annotation guidelines and annotator consensus, without an external gold standard or reader-response test.
  • domain assumption Adjudicated consensus labels are reliable ground truth despite low raw pairwise agreement.
    Section 3 and Tables 3-4: exact-match pairwise F1 for informational bias spans ranges from 12.68 to 20.68. The paper treats discussion-resolved labels as gold and reports higher agreement with the gold standard (0.70 informational span F1).
  • domain assumption The 100 selected triplets are representative enough to support claims about 'news articles' generally.
    Appendix B: 16 triplets were directly selected as highly visible, polarizing events, and the remaining triplets were manually curated from Common Crawl alignments. This is a convenience sample, not a random sample of news coverage.
  • domain assumption FOX, NYT, and HPO have the ideological positions assigned by Budak et al. (2016) throughout the 2010 to 2019 study window.
    Appendix B uses Budak et al. to label outlet ideology, and the entity-portrayal analysis depends on those labels. Outlet leanings could shift over the decade.
invented entities (1)
  • Informational bias as a formal annotation category
    purpose: To name and label factual content that is selected or backgrounded to sway opinion, as distinct from lexical bias.
    The category is defined and validated only within this paper's annotation framework; no external benchmark or downstream behavioral test is provided to show that labeled spans affect reader opinion beyond annotator judgment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In Plain Sight: Media Bias Through the Lens of Factual Reporting." pith.science (2026). https://pith.science/paper/BF4GYLQ2

@misc{pith2026190902670,
  author       = {Pith},
  title        = {Pith review of: In Plain Sight: Media Bias Through the Lens of Factual Reporting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BF4GYLQ2}},
  note         = {Machine review of arXiv:1909.02670}
}
read the original abstract

The increasing prevalence of political bias in news media calls for greater public awareness of it, as well as robust methods for its detection. While prior work in NLP has primarily focused on the lexical bias captured by linguistic attributes such as word choice and syntax, other types of bias stem from the actual content selected for inclusion in the text. In this work, we investigate the effects of informational bias: factual content that can nevertheless be deployed to sway reader opinion. We first produce a new dataset, BASIL, of 300 news articles annotated with 1,727 bias spans and find evidence that informational bias appears in news articles more frequently than lexical bias. We further study our annotations to observe how informational bias surfaces in news articles by different media outlets. Lastly, a baseline model for informational bias prediction is presented by fine-tuning BERT on our labeled data, indicating the challenges of the task and future directions.

Figures

Figures reproduced from arXiv: 1909.02670 by the authors.

Figure 1
Figure 1. Examples of negative bias from Huffington [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Distribution of lexical and informational bias [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Excerpts showing different types of informa [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Our Javascript annotation tool at various steps. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MGM: Global Understanding of Audience Overlap Graphs for Predicting the Factuality and the Bias of News Media

    cs.LG 2024-12 conditional novelty 6.0 of 10

    MGM augments graph neural networks with globally similar media nodes and language model probabilities, improving factuality and bias classification of news outlets.

Reference graph

Works this paper leans on

30 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ceren Budak, Sharad Goel, and Justin M Rao. 2016. Fair and balanced? quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly, 80(S1):250--271

  4. [4]

    Dallas Card, Amber E Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 438--444

  5. [5]

    Yoonjung Choi and Janyce Wiebe. 2014. +/-effectwordnet: Sense-level lexicon acquisition for opinion inference. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1181--1191

  6. [6]

    Claes De Vreese. 2004. The effects of strategic news on political cynicism, issue evaluations, and policy support: A two-wave experiment. Mass Communication & Society, 7(2):191--214

  7. [7]

    Stefano DellaVigna and Matthew Gentzkow. 2010. Persuasion: empirical evidence. Annu. Rev. Econ., 2(1):643--669

  8. [8]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171--4186

Show all 30 references
  1. [9]

    Robert M Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of communication, 43(4):51--58

  2. [10]

    Robert M Entman. 2007. Framing bias: Media in the distribution of power. Journal of communication, 57(1):163--173

  3. [11]

    Matthew Gentzkow and Jesse M Shapiro. 2006. Media bias and reputation. Journal of political Economy, 114(2):280--316

  4. [12]

    Matthew Gentzkow and Jesse M Shapiro. 2010. What drives media slant? evidence from us daily newspapers. Econometrica, 78(1):35--71

  5. [13]

    Matthew Gentzkow and Jesse M Shapiro. 2011. Ideological segregation online and offline. The Quarterly Journal of Economics, 126(4):1799--1839

  6. [14]

    Matthew Gentzkow, Jesse M Shapiro, and Daniel F Stone. 2015. Media bias in the marketplace: Theory. In Handbook of media economics, volume 1, pages 623--645. Elsevier

  7. [15]

    Stephan Greene and Philip Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In Proceedings of human language technologies: The 2009 annual conference of the north american chapter of the association for computational linguistics, pages 503--511. Associ...

  8. [16]

    Christoph Hube and Besnik Fetahu. 2019. Neural based statement classification for biased language. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 195--203. ACM

  9. [17]

    Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1113--1122

  10. [18]

    Maxwell McCombs and Amy Reynolds. 2009. How the news shapes our civic agenda. In Media effects, pages 17--32. Routledge

  11. [19]

    David Niven. 2001. Bias in the news: Partisanship and negativity in media coverage of presidents george bush and bill clinton. Harvard International Journal of Press/Politics, 6(3):31--46

  12. [20]

    Thomas E Patterson. 1996. Bad news, bad governance. The Annals of the American Academy of Political and Social Science, 546(1):97--108

  13. [21]

    Elizabeth M Perse. 2001. Media effects and society. Routledge

  14. [22]

    Andrea Prat and David Str \"o mberg. 2013. The political economy of mass media. Advances in economics and econometrics, 2:135

  15. [23]

    Markus Prior. 2013. Media and political polarization. Annual Review of Political Science, 16:101--127

  16. [24]

    Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1650--1659

  17. [25]

    Amy Reynolds and Maxwell McCombs. 2002. News influence on our pictures of the world. In Media effects, pages 11--28. Routledge

  18. [26]

    Swapna Somasundaran, Josef Ruppenhofer, and Janyce Wiebe. 2008. Discourse level opinion relations: An annotation study. In Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue, pages 129--137

  19. [27]

    Cigdem Toprak, Niklas Jakob, and Iryna Gurevych. 2010. Sentence and expression level annotation of opinions in user-generated discourse. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 575--584. Association for Computational Li...

  20. [28]

    Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165--210

  21. [29]

    Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing

  22. [30]

    Tae Yano, Philip Resnik, and Noah A Smith. 2010. Shedding (a thousand points of) light on biased language. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 152--158. Association for Computational Linguistics

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.