pith. sign in

arxiv: 1907.03718 · v1 · pith:RF4NQL7Ynew · submitted 2019-07-08 · 💻 cs.IR · cs.CY

CobWeb: A Research Prototype for Exploring User Bias in Political Fact-Checking

Anubrata Das , Kunjan Mehta , Matthew Lease This is my paper

Pith reviewed 2026-05-25 00:49 UTC · model grok-4.3

classification 💻 cs.IR cs.CY
keywords user biaspolitical fact-checkinguser interfacebias visualizationnews source reputationveracity judgmentuser study
0
0 comments X

The pith

A prototype shows users their estimated bias from news source preferences and 80% find the indicator useful for judging political claims.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents CobWeb, a research prototype that estimates user bias from how people rate the reputation of news sources and displays this estimate inside a fact-checking interface. The system aims to make users aware of how their own leanings may shape their assessment of a claim's truth. In a user study, 80% of participants reported that seeing the bias indicator helped them judge the veracity of political claims. The work focuses on communicating bias to users rather than on automated detection alone.

Core claim

We estimate the user bias as a function of the user's perceived reputation of the news sources (e.g., a user with liberal beliefs may tend to trust liberal sources). We build an interface to communicate the role of estimated user bias in the context of a fact-checking task. We also explore the utility of helping users visualize their detected level of bias. 80% of the users of our system find that the presence of an indicator for user bias is useful in judging the veracity of a political claim.

What carries the argument

CobWeb interface that estimates user bias from perceived news source reputations and visualizes the estimate during fact-checking tasks.

If this is right

  • An indicator of estimated user bias is perceived as useful by 80% of users when judging the veracity of political claims.
  • Bias can be modeled practically from users' stated trust in different news sources.
  • Visualizing detected bias supports users in assessing claim truthfulness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the self-reported usefulness holds in real settings, similar indicators could be added to existing fact-checking platforms to surface source-based leanings.
  • The method could be tested in other domains where source reputation influences judgment, such as health or consumer information.
  • Longitudinal studies might check whether repeated exposure to the bias display changes how users select or trust sources over time.

Load-bearing premise

User bias can be meaningfully estimated as a function of perceived reputation of news sources and self-reported usefulness in a prototype study reflects real improvement in fact-checking accuracy.

What would settle it

A controlled experiment in which users shown the bias indicator do not produce more accurate veracity judgments than a control group without the indicator.

Figures

Figures reproduced from arXiv: 1907.03718 by Anubrata Das, Kunjan Mehta, Matthew Lease.

Figure 1
Figure 1. Figure 1: The CoBWeb interface with claim and bias scores. Our interface extends Nguyen et al. [17]’s design and prototype. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
read the original abstract

The effect of user bias in fact-checking has not been explored extensively from a user-experience perspective. We estimate the user bias as a function of the user's perceived reputation of the news sources (e.g., a user with liberal beliefs may tend to trust liberal sources). We build an interface to communicate the role of estimated user bias in the context of a fact-checking task. We also explore the utility of helping users visualize their detected level of bias. 80% of the users of our system find that the presence of an indicator for user bias is useful in judging the veracity of a political claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper presents CobWeb, a research prototype that estimates user bias as a function of perceived news-source reputation and provides an interface to visualize this bias during political fact-checking tasks. It reports that 80% of users found the bias indicator useful for judging claim veracity.

Significance. If the empirical result holds under proper validation, the work would contribute a UX-focused exploration of bias mitigation in fact-checking interfaces, an area noted as underexplored. The prototype approach and direct description of the bias-estimation method are clear strengths for a systems paper.

major comments (3)
  1. [Abstract] Abstract: The headline claim that '80% of the users of our system find that the presence of an indicator for user bias is useful in judging the veracity of a political claim' is presented with no information on sample size, participant recruitment, study design, controls, or statistical tests. This absence makes the central empirical result impossible to assess for reliability or generalizability.
  2. [Evaluation / User Study] The evaluation relies exclusively on self-reported usefulness; no objective measures (pre/post accuracy on held-out claims, error rates, or calibration against ground truth) are reported to test whether the bias indicator actually improves veracity judgment. This leaves the functional claim unsupported.
  3. [System Description] The bias-estimation step (perceived reputation of news sources) is not validated against any external ground truth for user bias, so the mapping from perceived reputation to estimated bias remains an untested modeling choice.
minor comments (1)
  1. [Abstract] The abstract and introduction could more clearly distinguish between perceived usefulness and demonstrated improvement in fact-checking accuracy.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment below, indicating revisions where appropriate to improve clarity and scope without overstating the exploratory nature of the prototype and study.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The headline claim that '80% of the users of our system find that the presence of an indicator for user bias is useful in judging the veracity of a political claim' is presented with no information on sample size, participant recruitment, study design, controls, or statistical tests. This absence makes the central empirical result impossible to assess for reliability or generalizability.

    Authors: We agree that the abstract should include these details for proper context. In the revised manuscript we will expand the abstract to summarize the sample size, recruitment approach, and study design drawn from the evaluation section. As this was an exploratory prototype evaluation, no formal statistical tests or controls were applied. revision: yes

  2. Referee: [Evaluation / User Study] The evaluation relies exclusively on self-reported usefulness; no objective measures (pre/post accuracy on held-out claims, error rates, or calibration against ground truth) are reported to test whether the bias indicator actually improves veracity judgment. This leaves the functional claim unsupported.

    Authors: The study was intentionally scoped to measure perceived usefulness of the bias visualization in an early-stage systems prototype, which aligns with common practice for UX-focused exploration papers. Objective accuracy measures were not collected because the primary aim was to examine how users interact with and perceive the bias indicator rather than to demonstrate performance gains. We will revise the evaluation section to explicitly state this scope and add a limitations paragraph acknowledging the absence of objective metrics. revision: partial

  3. Referee: [System Description] The bias-estimation step (perceived reputation of news sources) is not validated against any external ground truth for user bias, so the mapping from perceived reputation to estimated bias remains an untested modeling choice.

    Authors: We will revise the system description to clearly label the bias estimation as a modeling choice that uses perceived source reputation as a proxy, without external validation. This framing will be presented as an untested assumption of the prototype, with discussion of its implications added to the limitations section. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical prototype study with direct user feedback

full rationale

The paper presents a system prototype and reports results from a user study (80% find bias indicator useful). No equations, derivations, fitted parameters, or predictions are present. The bias estimation is described conceptually from perceived source reputation, but no self-referential reduction, self-citation load-bearing, or renaming of results occurs. The central claim is a straightforward empirical percentage from self-reports, with no mathematical chain that collapses to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The paper is an empirical HCI prototype description with no mathematical free parameters, axioms, or invented entities; the bias estimation is described at a high level without formalization.

pith-pipeline@v0.9.0 · 5626 in / 1066 out tokens · 27762 ms · 2026-05-25T00:49:49.027087+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages

  1. [1]

    Dimitrios Bountouridis, Monica Marrero, Nava Tintarev, and Claudia Hauff

    work page

  2. [2]

    In SIGIR workshop on ExplainAble Recommendation and Search (EARS)

    Explaining Credibility in News Articles using Cross-Referencing. In SIGIR workshop on ExplainAble Recommendation and Search (EARS)

    work page

  3. [3]

    Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth

    work page

  4. [4]

    In The 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics (NAACL 2019)

    Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims. In The 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics (NAACL 2019) . Minneapolis, Minnesota. http: //www.cis.upenn.edu/~ccb/publications/discovering-diverse-perspectives.pdf 9https://moody.utexas.edu/research/center-media-...

    work page 2019

  5. [5]

    Henriette Cramer, Vanessa Evers, Satyan Ramlal, Maarten Van Someren, Lloyd Rutledge, Natalia Stash, Lora Aroyo, and Bob Wielinga. 2008. The effects of transparency on trust in and acceptance of a content-based art recommender. User Modeling and User-Adapted Interaction 18, 5 (2008), 455

    work page 2008

  6. [6]

    Fred D Davis. 1989. Perceived usefulness, perceived ease of use, and user accep- tance of information technology. MIS quarterly (1989), 319–340

    work page 1989

  7. [7]

    Robert Epstein and Ronald E Robertson. 2015. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences 112, 33 (2015), E4512–E4521

    work page 2015

  8. [8]

    Seth Flaxman, Sharad Goel, and Justin M Rao. 2016. Filter bubbles, echo chambers, and online news consumption. Public opinion quarterly 80, S1 (2016), 298–320

    work page 2016

  9. [9]

    Thomas Fossen and Joel Anderson. 2014. What’s the point of voting advice applications? Competing perspectives on democracy and citizenship. Electoral Studies 36 (2014), 244–251

    work page 2014

  10. [10]

    R Kelly Garrett. 2009. Echo chambers online?: Politically motivated selective expo- sure among Internet news users. Journal of Computer-Mediated Communication 14, 2 (2009), 265–285

    work page 2009

  11. [11]

    Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krish- namurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring personalization of web search. In Proceedings of the 22nd international conference on World Wide Web. ACM, 527–538

    work page 2013

  12. [12]

    Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 1113–1122

    work page 2014

  13. [13]

    Kathleen Hall Jamieson and Joseph N Cappella. 2008. Echo chamber: Rush Lim- baugh and the conservative media establishment . Oxford University Press

    work page 2008

  14. [14]

    Diane Kelly et al. 2009. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends ® in Information Retrieval 3, 1–2 (2009), 1–224

    work page 2009

  15. [15]

    Annie YS Lau and Enrico W Coiera. 2009. Can cognitive biases during consumer health information searches be reduced to improve decision making? Journal of the American Medical Informatics Association 16, 1 (2009), 54–65

    work page 2009

  16. [16]

    Matthew Lease. 2018. Fact Checking and Information Retrieval. (2018)

    work page 2018

  17. [17]

    Q Vera Liao and Wai-Tat Fu. 2013. Beyond the filter bubble: interactive effects of perceived threat and topic involvement on selective exposure to information. In Proceedings of the SIGCHI conference on human factors in computing systems . ACM, 2359–2368

    work page 2013

  18. [18]

    Sayooran Nagulendra and Julita Vassileva. 2014. Understanding and controlling the filter bubble through interactive visualization: a user study. In Proceedings of the 25th ACM conference on Hypertext and social media . ACM, 107–115

    work page 2014

  19. [19]

    An T Nguyen, Aditya Kharosekar, Saumyaa Krishnan, Siddhesh Krishnan, Eliza- beth Tate, Byron C Wallace, and Matthew Lease. 2018. Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking. In The 31st Annual ACM Symposium on User Interface Software and Technology . ACM, 189–199

    work page 2018

  20. [20]

    An T Nguyen, Aditya Kharosekar, Matthew Lease, and Byron C Wallace. 2018. An Interpretable Joint Graphical Model for Fact-Checking From Crowds.. In AAAI

    work page 2018

  21. [21]

    Eli Pariser. 2011. The filter bubble: What the Internet is hiding from you . Penguin UK

    work page 2011

  22. [22]

    Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum

    work page

  23. [23]

    CRF 71, 88.74 (2018), 80–00

    CredEye: A Credibility Lens for Analyzing and Explaining Misinformation. CRF 71, 88.74 (2018), 80–00

    work page 2018

  24. [24]

    Pearl Pu, Li Chen, and Rong Hu. 2011. A user-centric evaluation framework for recommender systems. InProceedings of the fifth ACM conference on Recommender systems. ACM, 157–164. 7

    work page 2011

  25. [25]

    Walter Quattrociocchi, Antonio Scala, and Cass R Sunstein. 2016. Echo chambers on Facebook. (2016)

    work page 2016

  26. [26]

    Paul Resnick, R Kelly Garrett, Travis Kriplean, Sean A Munson, and Natalie Jomini Stroud. 2013. Bursting your (filter) bubble: strategies for promoting diverse exposure. In Proceedings of the 2013 conference on Computer supported cooperative work companion. ACM, 95–100

    work page 2013

  27. [27]

    Elisa Shearer and Jeffrey Gottfried. 2017. News use across social media platforms

    work page 2017

  28. [28]

    Pew Research Center, Journalism and Media (2017)

    work page 2017

  29. [29]

    Rashmi Sinha and Kirsten Swearingen. 2002. The role of transparency in rec- ommender systems. In CHI’02 extended abstracts on Human factors in computing systems. ACM, 830–831

    work page 2002

  30. [30]

    Christian Stab, Johannes Daxenberger, Chris Stahlhut, Tristan Miller, Benjamin Schiller, Christopher Tauchmann, Steffen Eger, and Iryna Gurevych. 2018. Argu- menText: Searching for Arguments in Heterogeneous Sources. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Demonstrations. 21–25

    work page 2018

  31. [31]

    Nynke Tromp, Paul Hekkert, and Peter-Paul Verbeek. 2011. Design for socially responsible behavior: a classification of influence based on intended user experi- ence. Design Issues 27, 3 (2011), 3–19

    work page 2011

  32. [32]

    Frederik Zuiderveen Borgesius, Damian Trilling, Judith Moeller, Balázs Bodó, Claes H de Vreese, and Natali Helberger. 2016. Should we worry about filter bubbles? (2016). 8

    work page 2016