Pith. sign in

REVIEW 4 major objections 4 minor 27 references

Automatic Fact-Checking Using Context and Discourse Information

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automatic fact-checking performs best when it models conversational context, not just the wording of the claim.

desk verdict Solid empirical study, but the headline claim about context is not cleanly isolated by the ablations: the 'context' group bundles target-sentence content (and, in cQA, Web evidence) with genuinely contextual features. read the letter →

arxiv 1908.01328 v1 pith:7WX4WIUG submitted 2019-08-04 cs.CL cs.AI

Pepa Atanasova , Preslav Nakov , Lluís Màrquez , Alberto Barrón-Cedeño , Georgi Karadzhov , Tsvetomila Mihaylova , Mitra Mohtarami , James Glass This is my paper · ORCID
classification cs.CLcs.AI
keywords automaticfact-checkingcheck-worthyclaimdetectioncommunityquestionansweringdiscourseparsingcontextualfeaturespoliticaldebatesanswerverificationmulti-tasklearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that automatic fact-checking systems are meaningfully better when they model where a claim sits in a conversation, rather than treating the claim sentence in isolation. Across two tasks—deciding which statements in US presidential debates deserve fact-checking, and judging whether forum answers are factually true—context features improve ranking and accuracy more than any other feature group. Discourse structure, such as the rhetorical relations between sentences, also helps, but less than conversational context. If this is right, deployed fact-checkers should draw on surrounding turns, thread-level agreement, and related posts instead of verifying each sentence alone.

What carries the argument

The load-bearing mechanism is a feature design that embeds each target sentence in progressively wider rings of context: its position within a speaker turn, the previous and next turns, metadata about speaker and audience reaction, topic and embedding similarity to the surrounding segments, contradiction cues, and, for forum answers, similarity to other answers in the thread and to curated high-quality posts. Discourse context is supplied by a Rhetorical Structure Theory parser, a representation of text as a tree of nucleus–satellite relations, whose features count how the target sentence connects to its neighbors through relations such as Background, Elaboration, and Contrast. The feature-removal experiments—taking each group out and measuring the drop in mean average precision or accuracy—are what carry the argument, since they isolate the contribution of context and discourse.

What would settle it

Re-annotate the 249 forum answers with an independent protocol and authoritative sources, then rerun the feature-removal experiments; if removing context no longer produces the reported accuracy drop, the effect was label noise. Alternatively, shuffle or delete the surrounding debate turns while keeping labels fixed and check whether mean average precision still falls by the reported 4.2 points.

Watch

Extended reading notes

Core claim

The paper establishes that for check-worthiness detection, modeling the target sentence inside its speaker turn and against previous and following turns, speaker metadata, audience reactions, topic distributions, embedding similarity, and contradiction cues yields a ranking system that beats a re-implemented sentence-only baseline by a large margin. Removing the contextual features drops mean average precision from 0.427 to 0.385, while removing the discourse features drops it to 0.412. For answer fact-checking in a community forum, representing an answer against the rest of its thread, against high-quality posts, against the whole forum, and against web evidence reaches 0.683 accuracy; removing context drops that to 0.574 and removing discourse to 0.659. The authors read the consistent pattern across both tasks as evidence that context and discourse carry information that target-sentence features miss.

Load-bearing premise

The central premise is that the gold labels—which media fact-checkers selected and which four annotators judged as true or false using external websites—are correct for every item, so the measured gaps come from context information rather than from label mistakes.

Editorial extensions

If this is right

  • Fact-checking systems should treat a claim as a node in a conversation, not as an isolated sentence; conversational context is worth roughly 4 MAP points in debate claim ranking and about 11 accuracy points in forum answer verification.
  • Answer veracity in community forums is learnable separately from answer quality, since a 'good' answer can still be false and thread-level agreement is a usable signal for truthfulness.
  • Multi-task learning across fact-checking organizations improves the system's ability to mimic any single organization's check-worthiness choices for most of the nine organizations tested, suggesting that joint modeling of editorial policies is useful.
  • Discourse relations such as Background and Elaboration carry a smaller but non-zero signal, and combining them with context features produces the best overall performance in both tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not develop the point that the context effect may partly be social consensus: forum users tend to repeat correct answers and challenge false ones, so a simpler system based only on repeated-answer agreement might capture much of the same gain.
  • The discourse priors associated with true statements (Elaboration, Background) are learned from one debate and one forum domain; an obvious extension is to test whether those same relations transfer to news articles, social media, or scientific claims, where discourse conventions differ.
  • A clean test the paper does not run would hold the gold labels fixed and perturb only the context—shuffling debate turns or removing thread agreement—to quantify how much of the reported gain is genuinely contextual rather than lexical overlap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This manuscript studies the impact of contextual and discourse information on two fact-checking-related tasks: check-worthy claim identification in political debates and fact-checking of answers in community question answering. The authors develop supervised models combining neural networks, kernel SVMs, and a rich feature set; they create two annotated datasets (CW-USPD-2016 and CQA-QL-FACT); and they run ablation experiments that appear to show that removing context features degrades performance substantially, with discourse features also contributing. The paper additionally presents multi-task learning experiments for mimicking individual fact-checking organizations. The central conclusion is that discourse cues and especially contextual features play an important role in these tasks.

Significance. If the empirical conclusions are correct, the paper would provide a useful demonstration that conversational context and discourse structure improve automatic fact-checking systems, and the released datasets and code are valuable resources for the community. The feature engineering is careful, and the comparison to ClaimBuster provides a reasonable baseline. However, the load-bearing ablation evidence is confounded as discussed in the major comments, and the reporting inconsistencies need to be fixed before the results can be fully trusted. The paper's contribution is therefore conditional on additional analysis.

major comments (4)
  1. [Sections 2.2.4, 2.2.5, Table 4] The 'Topics' and 'Embeddings' feature groups each contain 300-dimensional representations of the target sentence itself, with only three additional cosine features for context. The 'All\context' ablation removes these entire groups, so the 4.2 MAP drop (from 0.427 to 0.385) cannot be attributed solely to the loss of context. To support the claim that context is especially important, the ablation should isolate the context-specific components (e.g., the cosines and the position, segment-size, and metadata features) while retaining the target-sentence representations.
  2. [Sections 3.2.1-3.2.3, 3.3.3, Table 12] The 'context' feature group for the fact-checking task includes support from the current thread, support from the entire Qatar Living forum, and support from high-quality posts, with the latter computed via web search (Section 3.3.3). The 'All\context' ablation therefore removes external evidence features as well as conversational context. The resulting accuracy drop (from 0.683 to 0.574) supports a broad notion of 'evidence use' rather than specifically conversational context. Please either restrict the ablation to thread-level context features or rephrase the conclusion to match the feature group actually removed.
  3. [Abstract and Section 1 vs. Section 3.5.2 and Table 12] The abstract and introduction state that discourse and context improve the fact-checking task by 4.5 MAP absolute points, but Table 12 reports Accuracy, with a gap of 0.141 between 'All information sources' (0.683) and 'All\discourse and context' (0.542). No MAP values are reported for this task. This misreporting affects the paper's headline result and should be corrected.
  4. [Tables 4 and 12] The main ablation results are reported without significance tests or confidence intervals. The check-worthiness evaluation uses only four cross-validation folds (one debate per fold) and the fact-checking evaluation uses 71 leave-one-thread-out folds; observed differences could be within variance. Provide bootstrap or permutation-based confidence intervals for the key comparisons, or at least report variance across folds.
minor comments (4)
  1. [Section 2.2.4] The 'current segment' cosine feature is computed against a segment that contains the target sentence, so it may reflect self-similarity. Please clarify this or remove the feature when claiming a context contribution.
  2. [Section 3.1] Please report inter-annotator agreement for the question factuality labels and the answer veracity labels; the current description mentions four annotators but gives no agreement statistics, which would strengthen confidence in the gold standard.
  3. [Table 4] The row 'Only context+discourse' has MAP 0.317, which equals Claimbuster-Platform's MAP; the comparison would be clearer if the text noted that this is achieved without the ClaimBuster features.
  4. [Section 2.4] The neural network hyperparameters (300 epochs, batch size 550) appear to be chosen without an explicit early-stopping or validation-based criterion; please mention whether overfitting was observed or how the hyperparameters were selected.

Circularity Check

1 steps flagged · score 4.0 of 10

No self-citation chain; the central context claim is partly undermined by a definitional conflation in the context feature group.

  1. self definitional [Sections 2.2.4-2.2.5 and Table 4]
    "We had 300 topics, and we used the distribution over the topics as a representation for the target sentence. We further modeled the context using cosines with such representations for the previous, the current, and the next segment. ... We used the pre-trained 300-dimensional Google News word embeddings ... to compute an average embedding vector for the target sentence, and we used the 300 dimensions of that vector. We also modeled the context as the cosine between that vector and the vectors for three segments: the previous, the current, and the following one."

    The feature groups called 'context' in the check-worthiness system each contain a full target-sentence representation (300 LDA topic dimensions and 300 average-embedding dimensions), not only surrounding context. The 'All\context' row of Table 4 removes these groups wholesale, so the reported MAP drop from 0.427 to 0.385 cannot be attributed to context: by construction, removing 'context' also removes the target sentence's own content. Moreover, the 'current segment' cosine is computed against a segment that includes the target sentence, adding self-similarity. The conclusion that 'especially modeling the context' is important is thus partly a consequence of defining 'context' to include the target, rather than an isolated measurement of context.

full rationale

The paper is otherwise empirically self-contained: datasets are anchored to external annotations (nine fact-checking organizations; SemEval-2016 CQA-QL-2016), evaluation uses held-out debates and leave-one-thread-out folds, and no uniqueness theorem or load-bearing self-citation forces the choice of architecture. The identified issue is a definitional/ablation confound in the central claim of check-worthiness: the Topics and Embeddings groups include target-sentence features, so the All\context drop conflates content and context; the current-segment cosine also includes the target. For the fact-checking task, the 'context' group similarly bundles evidence retrieved from the Web/forum, so it measures broad evidence use rather than conversational context. These problems weaken the stated conclusion that context is especially important, but they do not make the whole derivation equivalent to its inputs; the released datasets and baselines remain independently testable.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

This is an empirical NLP paper; no mathematical derivation or invented entities. The main loaded assumptions concern annotation quality and the reliability of external feature extractors. The free parameters are the usual machine-learning hyperparameters, which were tuned on the training folds and are not central to the scientific claim.

assumptions (2)
  • domain assumption The gold annotations (fact-checking by nine media sources; manual factuality labels for 249 forum answers) are correct and reliable.
    The entire evaluation treats these labels as ground truth; errors or subjectivity in annotation would directly bias the measured effect of context and discourse features. See Sections 2.1 and 3.1.
  • domain assumption The discourse parser (CODRA) and the external resources (LDA topics, Google News embeddings, Bing/Google search) provide sufficiently accurate signals for the features to be meaningful.
    Feature quality depends on these tools; parser errors would weaken the discourse features. See Sections 2.2.8 and 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Fact-Checking Using Context and Discourse Information." pith.science (2026). https://pith.science/paper/7WX4WIUG

@misc{pith2026190801328,
  author       = {Pith},
  title        = {Pith review of: Automatic Fact-Checking Using Context and Discourse Information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7WX4WIUG}},
  note         = {Machine review of arXiv:1908.01328}
}
read the original abstract

We study the problem of automatic fact-checking, paying special attention to the impact of contextual and discourse information. We address two related tasks: (i) detecting check-worthy claims, and (ii) fact-checking claims. We develop supervised systems based on neural networks, kernel-based support vector machines, and combinations thereof, which make use of rich input representations in terms of discourse cues and contextual features. For the check-worthiness estimation task, we focus on political debates, and we model the target claim in the context of the full intervention of a participant and the previous and the following turns in the debate, taking into account contextual meta information. For the fact-checking task, we focus on answer verification in a community forum, and we model the veracity of the answer with respect to the entire question--answer thread in which it occurs as well as with respect to other related posts from the entire forum. We develop annotated datasets for both tasks and we run extensive experimental evaluation, confirming that both types of information ---but especially contextual features--- play an important role.

Figures

Figures reproduced from arXiv: 1908.01328 by the authors.

Figure 1
Figure 1. The architecture of the full neural multi-source learning model, predicting the selection choices of each of the nine individual [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Example from the Qatar Living forum. We can see that the model any performs comparably to the singleton baseline, thus being clearly inferior than the multi-task learning variants. Finally, singleton+any is also better than the single-task learning variants, but it falls short compared to the other multi-task learning variants. Including output units for all nine individual media seems crucial for getting advantage … view at source ↗
Figure 3
Figure 3. Our NN architecture for fact-checking in cQA. Each green box in [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Examples of the impact of context in debate instances. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Examples of the impact of context and discourse in cQA instances. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 26 canonical work pages

  1. [2]

    In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT ’18)

    Integrating Stance Detection and Fact Checking in a Unified Corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT ’18). New Orleans, Louisiana, USA, 21–27. Protima Banerjee and Hyoil Han

    work page 2018

  2. [3]

    In CLEF 2018 Working Notes

    Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims, Task 2: Factuality. In CLEF 2018 Working Notes. Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum (CEUR Workshop Proceedings) . Avignon, France. Jonathan Baxter

    work page 2018

  3. [13]

    Science 359, 6380 (2018), 1094–1096

    The Science of Fake News. Science 359, 6380 (2018), 1094–1096. Dieu-Thu Le, Ngoc Thang Vu, and Andre Blessing

    work page 2018

  4. [15]

    In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing (EMNLP ’01)

    Detecting Short Passages of Similar Text in Large Document Collections. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing (EMNLP ’01) . Pittsburgh, Pennsylvania, USA. Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha

    work page 2001

  5. [16]

    Internet Research 28, 5 (2018), 1292–1312

    The Dark Side of News Community Forums: Opinion Manipulation Trolls. Internet Research 28, 5 (2018), 1292–1312. Todor Mihaylov and Preslav Nakov. 2016a. Hunting for Troll Comments in News Community Forums. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL ’16) . Berlin, Germany, 399–405. Todor Mihaylov and Pre...

    work page 2018

  6. [17]

    In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI ’18)

    Fact Checking in Community Forums. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI ’18) . New Orleans, Lousiana, USA, 5309–5316. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013a. Exploiting Similarities among Languages for Machine Translation. CoRR abs/1309.4168 (2013). Tomas Mikolov, Wen-tau Yih, and Geoffrey Zwei...

    arXiv 2013

  7. [18]

    29, 3 (2013), 436–465

    Crowdsourcing a Word-Emotion Association Lexicon. 29, 3 (2013), 436–465. Mitra Mohtarami, Ramy Baly, James Glass, Preslav Nakov, Lluís Màrquez, and Alessandro Moschitti

    work page 2013

  8. [19]

    Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In Proceedings of the Ninth International Conference of the CLEF Association: Experimental IR Meets Multilinguality, Multimodality, and Interaction (Lecture Notes in Computer Science). Springer, Avignon, France, 372–387. Preslav Nakov, Doris Hoogevee...

    work page 2018

Show all 27 references
  1. [20]

    In Proceedings of the International Workshop on Semantic Evaluation (SemEval ’15)

    SemEval-2015 Task 3: Answer Selection in Community Question Answering. In Proceedings of the International Workshop on Semantic Evaluation (SemEval ’15) . Denver, Colorado, USA, 269–281. Preslav Nakov, Lluís Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhak...

    2015

  2. [22]

    ACM Trans

    Overview of the Special Issue on Trust and Veracity of Information in Social Media. ACM Trans. Inf. Syst. 34, 3 (2016), 14:1–14:5. Ayush Patwari, Dan Goldwasser, and Saurabh Bagchi

    2016

  3. [23]

    In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM)

    TATHYA: A Multi-Classifier System for Detecting Check-Worthy Statements in Political Debates. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM) . Singapore, 2259–2262. Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum

    2017

  4. [25]

    In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing (EMNLP ’03)

    Learning Extraction Patterns for Subjective Expressions. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing (EMNLP ’03) . Sapporo, Japan, 105–112. Qi Su, Helen Kai-Yun Chen, and Chu-Ren Huang

    2003

  5. [26]

    Science 359, 6380 (2018), 1146–1151

    The Spread of True and False News Online. Science 359, 6380 (2018), 1146–1151. Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie

    2018

  6. [27]

    PLoS ONE 11, 3 (03 2016), 1–29

    Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads. PLoS ONE 11, 3 (03 2016), 1–29. Manuscript submitted to ACM

    2016

  7. [1971]

    Language 47, 2 (1971), 340–358

    Implicative Verbs. Language 47, 2 (1971), 340–358. Milen Kouylekov and Matteo Negri

    1971

  8. [1997]

    Machine Learning 28, 1 (1997), 7–39

    A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling. Machine Learning 28, 1 (1997), 7–39. David M Blei, Andrew Y Ng, and Michael I Jordan

    1997

  9. [1998]

    Gradient-Based Learning Applied to Document Recognition. Proc. IEEE 86, 11 (1998), 2278–2324. Lucian Vlad Lita, Andrew Hazen Schlaikjer, WeiChang Hong, and Eric Nyberg

    1998

  10. [2001]

    Newspaper Research Journal 22, 2 (2001),

    Online Journalists Embrace New Marketing Function. Newspaper Research Journal 22, 2 (2001),

    2001

  11. [2003]

    Journal of Machine Learning Research 3, 1 (2003), 993–1022

    Latent Dirichlet Allocation. Journal of Machine Learning Research 3, 1 (2003), 993–1022. Ann M Brill

    2003

  12. [2007]

    In Proceedings of the 16th ACM Conference on Conference on Information and Knowledge Management (CIKM ’07)

    Discovering Authorities in Question Answer Communities by Using Link Analysis. In Proceedings of the 16th ACM Conference on Conference on Information and Knowledge Management (CIKM ’07) . Lisbon, Portugal, 919–922. Georgi Karadzhov, Pepa Gencheva, Preslav Nakov, and Ivan Koych...

    2017

  13. [2009]

    Natural Language Engineering 15, 4 (2009), i–xvii

    Recognizing Textual Entailment: Rational, Evaluation and Approaches. Natural Language Engineering 15, 4 (2009), i–xvii. Kareem Darwish, Dimitar Alexandrov, Preslav Nakov, and Yelena Mejova

    2009

  14. [2010]

    In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks

    Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks . Valletta, Malta, 45–50. Ellen Riloff and Janyce Wiebe

    2010

  15. [2013]

    InProceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM ’13)

    Battling the Internet Water Army: Detection of Hidden Paid Posters. InProceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM ’13) . Niagara, Ontario, Canada, 116–120. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth

    2013

  16. [2015]

    CODRA: A Novel Discriminative Framework for Rhetorical Analysis. Comput. Linguist. 41, 3 (2015), 385–435. Pawel Jurczyk and Eugene Agichtein

    2015

  17. [2016]

    In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval ’16)

    SemEval-2016 Task 3: Community Question Answering. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval ’16) . San Diego, California, 525–545. Preslav Nakov, Tsvetomila Mihaylova, Lluís Màrquez, Yashkumar Shiroya, and Ivan Koychev. 2017b. Do Not Tru...

    2016

  18. [2017]

    In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval ’17)

    SemEval-2017 Task 8: RumourEval: Determining Rumour Veracity and Support for Rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval ’17) . Vancouver, Canada, 60–67. Rob Ennals, Dan Byler, John Mark Agosta, and Barbara Rosario. 2010a. What is...

    2017

  19. [2018]

    In CLEF 2018 Working Notes

    Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims, Task 1: Check-Worthiness. In CLEF 2018 Working Notes. Avignon, France. Mouhamadou Lamine Ba, Laure Berti-Equille, Kushal Shah, and Hossam M. Hammady

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.