REVIEW 2 major objections 4 minor 1 cited by
In Plain Sight: Media Bias Through the Lens of Factual Reporting
T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Bias in news is mostly in selected facts, not loaded words
desk verdict A genuinely useful annotated resource for a real blind spot in bias detection, with a caveat that its headline prevalence claim rests on annotation discussions that start from very low agreement. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing artifact is BASIL, a span-level annotation dataset built from 100 article triplets, each triplet reporting the same event in Fox News, the New York Times, and Huffington Post. Annotators, blind to media source, mark every biased span as either lexical (removable or rephrasable without changing meaning) or informational (removable only by changing content or context), with additional labels for target, polarity, aim, and whether the span appears in a quote. The triplet design is essential: comparing three reports of the same event lets annotators distinguish event-relevant facts from peripheral details that shade opinion, and lets the paper measure outlet-level patterns while holding the story constant. The same structure powers the main baseline, a BERT model fine-tuned per bias type.
What would settle it
Re-annotate the same 100 article triplets with two groups of annotators of opposite political leanings who do not discuss or see outlet names, then compare the percentage of informational versus lexical spans. If the 72/28 split changes substantially by group ideology—or if the same article yields systematically different counts of informational spans—the prevalence finding would be an artifact of the annotation process rather than a stable property of news writing.
Extended reading notes
Core claim
The central discovery is that informational bias—sentences or clauses conveying tangential, speculative, or background facts that shape opinion about a political entity—appears in news more frequently than lexical bias. Across 1,727 annotated bias spans in the BASIL dataset, 1,249 (72.3%) are informational and 478 (27.7%) are lexical, a pattern consistent across all three outlets. While lexical bias clusters in the opening quarter of articles, informational bias is spread through the whole text, and almost half of it arrives inside quotations, suggesting outlets use selected quotes as a proxy for their own stance. In entity-level analysis, negative informational coverage tracks the outlet's ideology: conservative entities receive more negative informational bias in NYT and HPO, and liberal entities receive more in FOX.
Load-bearing premise
The central claim depends on the post-discussion labels reflecting recoverable properties of the text rather than the annotators' shared political beliefs; raw pairwise agreement on the exact text spans was low (roughly 13 to 21 on a 0 to 100 F1 scale), and the paper defers study of how annotators' prior beliefs affect their judgments.
Editorial extensions
If this is right
- Bias detection systems that only look at word choice will miss most bias; modeling content selection and article-level context is necessary.
- Quotations are a major channel for informational bias, so the editorial choice of which quotes to include should be treated as a strategic bias signal.
- Lexical bias tends to appear early in articles while informational bias is distributed throughout, so discourse position can help distinguish the two types.
- Span-level annotations reveal outlet slant toward specific political figures, not just an overall ideological leaning, enabling more granular media analysis.
- The sentence-level TF-IDF baseline's modest performance suggests that detecting informational bias requires information beyond the sentence being judged.
Reading between the lines
- If the prevalence finding holds, media literacy guidance should teach readers to ask what facts were included, omitted, or placed in quotes, not only to scan for emotionally charged words.
- The triplet-comparison method itself could be turned into a detection strategy: flag factual details that appear in only one outlet's account of the same event as candidate informational bias.
- Because raw pairwise annotator agreement on spans is low and the paper defers the study of annotators' prior beliefs, a replication with politically balanced annotator groups would test whether the 72/28 split is a stable property of news text or partly a product of the annotation process.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BASIL, a dataset of 300 news articles organized as 100 triplets (Fox News, New York Times, Huffington Post) reporting the same event, annotated at the span level for two types of bias: lexical bias (bias from word choice and syntax) and informational bias (factual content selected or contextualized to sway opinion). The authors report that informational bias spans outnumber lexical bias spans in all three outlets (72.3% vs. 27.7% overall), analyze differences in positional distribution, quote usage, and entity portrayal, and benchmark BERT and rule-based baselines for sentence-level and token-level bias detection. The central claim is that informational bias is more prevalent than lexical bias and that this finding motivates a shift in bias-detection research from surface linguistic cues to content selection and context.
Significance. If the prevalence claim is sound, the paper makes a meaningful contribution by refocusing media-bias research on content-level framing rather than lexical markers, and the BASIL dataset itself is a valuable resource with fine-grained annotations (bias type, target, polarity, aim, quote status). The experiments are honestly reported with 10-fold cross-validation, variance ranges, and a publicly available annotation tool. The descriptive statistics and entity-portrayal analysis offer a rich view of outlet-specific bias. However, the load-bearing prevalence claim depends entirely on the validity of a gold standard produced by two annotators who resolve large initial disagreements through discussion; the paper's own IAA numbers show raw span agreement is very low, and the reported improvement to gold-standard agreement does not demonstrate external validity. This makes the main conclusion provisional and in need of independent validation.
major comments (2)
- [Section 3, Table 3] The central claim that informational bias outnumbers lexical bias is a tally of gold-standard labels, but the gold standard is generated by two annotators who first disagree substantially (exact-match span F1 for informational bias ranges from 12.68 to 20.68, and lenient lexical F1 is at most 15.38) and then resolve disagreements via discussion. The paper states that individual-to-gold F1 rises to 0.70 (informational) and 0.56 (lexical), but because the gold standard is produced by the same two annotators, this only demonstrates convergence to a shared interpretation, not that the shared interpretation tracks recoverable text properties. Footnote 2 explicitly postpones study of annotators' prior beliefs, leaving open the possibility that the discussion process systematically inflates the novel 'informational' category relative to 'lexical.' To support the prevalence claim, the authors should provide independent evidence: for example, a fresh-team re-annotation of a sample, an adjudication protocol showing that disagreements are resolved by reference to textual criteria external to the annotators, or an analysis of how disagreement resolution shifts the type distribution. Without such evidence, the headline 72.3% versus 27.7% split could be an artifact of the annotation procedure.
- [Section 5, footnote 5] The TF-IDF baseline selects 'the 4 sentences in each article with the lowest average TF-IDF token scores,' with the justification that 'BASIL averages 4.1 informational bias spans per article.' This calibrates the baseline's only free parameter (k=4) directly to the gold-standard mean, giving the baseline oracle information about the number of spans. The comparison between this baseline and BERT is therefore not a fair parameter-free baseline; the claim that the baseline 'performs relatively well' and 'indicat[es] the importance of explicitly modeling context' is weakened because k is chosen to fit the dataset. The authors should either report results for several k values (e.g., k=2, 4, 6, 8) or remove the dataset-dependent calibration and discuss the sensitivity of the conclusion.
minor comments (4)
- [Section 4.1, Figure 2] The claim that 'lexical bias appears early in an article' and 'informational bias ... can appear at any position' is based on visual inspection of Figure 2; a statistical test (e.g., a chi-square or permutation test on quartile distributions) would make this quantitative claim more rigorous.
- [Section 3, Table 3] In Table 3, the 'B + C' row for lexical bias has dashes for exact matching because no exact lexical spans were shared; this is fine, but the paper should explicitly state whether the lenient-row F1=15.38 for B+C is based on one or few matches, given the small number of resolved lexical spans for that pair.
- [Section 5, Table 2] The sentence reports that the standard deviation across BERT folds ranges from 3.36 to 12.44; it would be clearer to report the standard deviation for each model in Table 2 rather than a global minimum and maximum, since some of the central comparisons (e.g., informational vs. lexical sentence F1) may overlap within noise.
- [Abstract and Figure 1] The abbreviation 'HPO' is used in Figure 1 and throughout, but it is only expanded as 'Huffington Post' in the text; consider defining it at first use in the caption or text for readers of the standalone figure.
Circularity Check
No significant circularity: the prevalence claim is a direct annotation tally, and no load-bearing step reduces to its own inputs.
full rationale
The paper's central descriptive claim—that informational bias spans outnumber lexical spans (Table 1)—is a direct count over the BASIL gold annotations, not the output of a fitted model or a theorem derived from the annotation definitions. The definition of informational bias (sentences/clauses that are tangential, speculative, or background) and lexical bias (bias from word choice and syntax) does not logically fix the span-count ratio; annotators could in principle have found more lexical spans. The only place where a quantity is calibrated to the gold data is the TF-IDF baseline, which chooses k=4 sentences per article because 'BASIL averages 4.1 informational bias spans per article' (footnote 5). This is explicitly a simple baseline for the detection experiment, not a prediction supporting the prevalence claim, and its reported sentence-level F1 of 26.02 is presented as a low benchmark rather than as evidence for the prevalence result. No load-bearing argument relies on self-citation: the cited prior work is external (e.g., Budak et al., Card et al., Recasens et al.), and no uniqueness result or ansatz is imported from the present authors' own prior papers. The acknowledged limitation (footnote 2: 'The likely effect of annotators' prior beliefs on their perception of bias will be investigated in future work') and the low raw span-level IAA (Table 3) are threats to the external validity of the annotation-based findings, but they do not make the derivation circular. The BERT experiments are independent benchmarks whose scores are reported as task-difficulty evidence, not renamed as confirmations of the prevalence claim. Hence no circular step is present.
Assumptions & free parameters
free parameters (1)
- k, number of low-TF-IDF sentences selected by the baseline =
4
assumptions (4)
- domain assumption Informational bias, defined as 'sentences or clauses that convey information tangential, speculative, or as background to the main event in order to sway readers' opinions', is a real and identifiable property of text.
- domain assumption Adjudicated consensus labels are reliable ground truth despite low raw pairwise agreement.
- domain assumption The 100 selected triplets are representative enough to support claims about 'news articles' generally.
- domain assumption FOX, NYT, and HPO have the ideological positions assigned by Budak et al. (2016) throughout the 2010 to 2019 study window.
invented entities (1)
-
Informational bias as a formal annotation category
Cite this review
Pith. "Pith review of In Plain Sight: Media Bias Through the Lens of Factual Reporting." pith.science (2026). https://pith.science/paper/BF4GYLQ2
@misc{pith2026190902670,
author = {Pith},
title = {Pith review of: In Plain Sight: Media Bias Through the Lens of Factual Reporting},
year = {2026},
howpublished = {\url{https://pith.science/paper/BF4GYLQ2}},
note = {Machine review of arXiv:1909.02670}
}
read the original abstract
The increasing prevalence of political bias in news media calls for greater public awareness of it, as well as robust methods for its detection. While prior work in NLP has primarily focused on the lexical bias captured by linguistic attributes such as word choice and syntax, other types of bias stem from the actual content selected for inclusion in the text. In this work, we investigate the effects of informational bias: factual content that can nevertheless be deployed to sway reader opinion. We first produce a new dataset, BASIL, of 300 news articles annotated with 1,727 bias spans and find evidence that informational bias appears in news articles more frequently than lexical bias. We further study our annotations to observe how informational bias surfaces in news articles by different media outlets. Lastly, a baseline model for informational bias prediction is presented by fine-tuning BERT on our labeled data, indicating the challenges of the task and future directions.
Figures
Forward citations
Cited by 1 Pith paper
-
MGM: Global Understanding of Audience Overlap Graphs for Predicting the Factuality and the Bias of News Media
MGM augments graph neural networks with globally similar media nodes and language model probabilities, improving factuality and bias classification of news outlets.
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Ceren Budak, Sharad Goel, and Justin M Rao. 2016. Fair and balanced? quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly, 80(S1):250--271
2016
-
[4]
Dallas Card, Amber E Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 438--444
work page 2015
-
[5]
Yoonjung Choi and Janyce Wiebe. 2014. +/-effectwordnet: Sense-level lexicon acquisition for opinion inference. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1181--1191
work page 2014
-
[6]
Claes De Vreese. 2004. The effects of strategic news on political cynicism, issue evaluations, and policy support: A two-wave experiment. Mass Communication & Society, 7(2):191--214
work page 2004
-
[7]
Stefano DellaVigna and Matthew Gentzkow. 2010. Persuasion: empirical evidence. Annu. Rev. Econ., 2(1):643--669
work page 2010
-
[8]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171--4186
2019
Show all 30 references
-
[9]
Robert M Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of communication, 43(4):51--58
1993
-
[10]
Robert M Entman. 2007. Framing bias: Media in the distribution of power. Journal of communication, 57(1):163--173
2007
-
[11]
Matthew Gentzkow and Jesse M Shapiro. 2006. Media bias and reputation. Journal of political Economy, 114(2):280--316
2006
-
[12]
Matthew Gentzkow and Jesse M Shapiro. 2010. What drives media slant? evidence from us daily newspapers. Econometrica, 78(1):35--71
2010
-
[13]
Matthew Gentzkow and Jesse M Shapiro. 2011. Ideological segregation online and offline. The Quarterly Journal of Economics, 126(4):1799--1839
2011
-
[14]
Matthew Gentzkow, Jesse M Shapiro, and Daniel F Stone. 2015. Media bias in the marketplace: Theory. In Handbook of media economics, volume 1, pages 623--645. Elsevier
2015
-
[15]
Stephan Greene and Philip Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In Proceedings of human language technologies: The 2009 annual conference of the north american chapter of the association for computational linguistics, pages 503--511. Associ...
2009
-
[16]
Christoph Hube and Besnik Fetahu. 2019. Neural based statement classification for biased language. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 195--203. ACM
2019
-
[17]
Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1113--1122
2014
-
[18]
Maxwell McCombs and Amy Reynolds. 2009. How the news shapes our civic agenda. In Media effects, pages 17--32. Routledge
2009
-
[19]
David Niven. 2001. Bias in the news: Partisanship and negativity in media coverage of presidents george bush and bill clinton. Harvard International Journal of Press/Politics, 6(3):31--46
2001
-
[20]
Thomas E Patterson. 1996. Bad news, bad governance. The Annals of the American Academy of Political and Social Science, 546(1):97--108
1996
-
[21]
Elizabeth M Perse. 2001. Media effects and society. Routledge
2001
-
[22]
Andrea Prat and David Str \"o mberg. 2013. The political economy of mass media. Advances in economics and econometrics, 2:135
2013
-
[23]
Markus Prior. 2013. Media and political polarization. Annual Review of Political Science, 16:101--127
2013
-
[24]
Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1650--1659
2013
-
[25]
Amy Reynolds and Maxwell McCombs. 2002. News influence on our pictures of the world. In Media effects, pages 11--28. Routledge
2002
-
[26]
Swapna Somasundaran, Josef Ruppenhofer, and Janyce Wiebe. 2008. Discourse level opinion relations: An annotation study. In Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue, pages 129--137
2008
-
[27]
Cigdem Toprak, Niklas Jakob, and Iryna Gurevych. 2010. Sentence and expression level annotation of opinions in user-generated discourse. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 575--584. Association for Computational Li...
2010
-
[28]
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165--210
2005
-
[29]
Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing
2005
-
[30]
Tae Yano, Philip Resnik, and Noah A Smith. 2010. Shedding (a thousand points of) light on biased language. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 152--158. Association for Computational Linguistics
2010
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.