{"id":"cd00e766-9677-485b-818a-0163923b6a61","arxiv_id":"1908.01328","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Adding contextual and discourse features improves automatic systems for spotting check-worthy claims in debates and for verifying facts in community forum answers.","lead":"This paper builds computer systems that find claims worth fact-checking in political debates and that check whether forum answers are true, using the surrounding conversation as a clue. The authors create new datasets and show that context and discourse features improve accuracy over standard sentence-only methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The check-worthiness 'context' ablation bundles the target sentence's own LDA and embedding vectors with true context features, so the reported 4.2-MAP drop does not isolate context.","rationale":"The reader's weakest assumption concerns the correctness of the CQA-QL-FACT gold labels. That is a legitimate data-quality issue, but it is secondary: even with perfect labels, the check-worthiness ablation does not isolate context because the 'Topics' and 'Embeddings' feature groups bundle target-sentence content with context cosines. The paper describes these groups under 'Modeling Context and Discourse', and Table 4 removes whole groups for the 'All\\context' row, so the 4.2-MAP drop cannot be cleanly attributed to context. I also note the lack of significance tests: with 249 answers, the Table 12 discourse effect (2.4 accuracy points) is within binomial noise, though the context effect is larger. I still would not reject the paper: the datasets and code are released, the fact-checking task shows a substantial context-group effect, and the qualitative examples provide some support. The right fix is a clean ablation, which is already a natural condition for acceptance. Hence the verdict remains UNCHANGED, but the condition should explicitly require separating target-content features from genuine context features in the check-worthiness experiment.","tokens_in":25762,"tokens_out":8377,"duration_ms":88150,"concrete_test":"Re-run Table 4 keeping the target-sentence LDA vector (Section 2.2.4) and target-sentence embedding (Section 2.2.5) in all rows, and remove only the true context sub-features: position (2.2.1), segment sizes (2.2.2), metadata (2.2.3), contradiction-context counts, known-example similarity (2.2.7), and the previous/next/current context cosines, recomputing the current-segment cosine after excluding the target sentence from the segment. If the resulting MAP drop is materially smaller than 0.042 (e.g., below 0.01), the reported contextual contribution is confounded with target content; if the drop remains large, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2.4 ('Topics', 303 features) and Section 2.2.5 ('Embeddings', 303 features) are labeled context features, but each includes a full target-sentence representation (300-dim LDA topic vector; 300-dim average word embedding) plus three context-cosine features. Table 4's 'All\\context' row removes these groups wholesale, so the MAP drop from 0.427 to 0.385 may reflect loss of target-sentence content, not loss of context. The same ablation also removes genuinely contextual position, segment-size, metadata, contradiction, and similarity features, but the bundling means the effect cannot be attributed to context. Additionally, the 'current segment' cosine is computed against a segment that contains the target sentence, so it can encode self-similarity. This is not a question of label noise or external-source authority; it is an internal confound in the experiment that is supposed to establish the paper's central claim that context is especially important. The fact-checking experiment in Table 12 avoids this particular target-content confound, but there the 'context' group includes Web/forum evidence features, so that result also speaks to broad evidence use rather than conversational context. A clean ablation is needed before the headline conclusion can be accepted as stated.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript studies the impact of contextual and discourse information on two fact-checking-related tasks: check-worthy claim identification in political debates and fact-checking of answers in community question answering. The authors develop supervised models combining neural networks, kernel SVMs, and a rich feature set; they create two annotated datasets (CW-USPD-2016 and CQA-QL-FACT); and they run ablation experiments that appear to show that removing context features degrades performance substantially, with discourse features also contributing. The paper additionally presents multi-task learning experiments for mimicking individual fact-checking organizations. The central conclusion is that discourse cues and especially contextual features play an important role in these tasks.","tokens_in":25958,"tokens_out":7648,"duration_ms":68219,"significance":"If the empirical conclusions are correct, the paper would provide a useful demonstration that conversational context and discourse structure improve automatic fact-checking systems, and the released datasets and code are valuable resources for the community. The feature engineering is careful, and the comparison to ClaimBuster provides a reasonable baseline. However, the load-bearing ablation evidence is confounded as discussed in the major comments, and the reporting inconsistencies need to be fixed before the results can be fully trusted. The paper's contribution is therefore conditional on additional analysis.","major_comments":[{"comment":"The 'Topics' and 'Embeddings' feature groups each contain 300-dimensional representations of the target sentence itself, with only three additional cosine features for context. The 'All\\context' ablation removes these entire groups, so the 4.2 MAP drop (from 0.427 to 0.385) cannot be attributed solely to the loss of context. To support the claim that context is especially important, the ablation should isolate the context-specific components (e.g., the cosines and the position, segment-size, and metadata features) while retaining the target-sentence representations.","section":"Sections 2.2.4, 2.2.5, Table 4"},{"comment":"The 'context' feature group for the fact-checking task includes support from the current thread, support from the entire Qatar Living forum, and support from high-quality posts, with the latter computed via web search (Section 3.3.3). The 'All\\context' ablation therefore removes external evidence features as well as conversational context. The resulting accuracy drop (from 0.683 to 0.574) supports a broad notion of 'evidence use' rather than specifically conversational context. Please either restrict the ablation to thread-level context features or rephrase the conclusion to match the feature group actually removed.","section":"Sections 3.2.1-3.2.3, 3.3.3, Table 12"},{"comment":"The abstract and introduction state that discourse and context improve the fact-checking task by 4.5 MAP absolute points, but Table 12 reports Accuracy, with a gap of 0.141 between 'All information sources' (0.683) and 'All\\discourse and context' (0.542). No MAP values are reported for this task. This misreporting affects the paper's headline result and should be corrected.","section":"Abstract and Section 1 vs. Section 3.5.2 and Table 12"},{"comment":"The main ablation results are reported without significance tests or confidence intervals. The check-worthiness evaluation uses only four cross-validation folds (one debate per fold) and the fact-checking evaluation uses 71 leave-one-thread-out folds; observed differences could be within variance. Provide bootstrap or permutation-based confidence intervals for the key comparisons, or at least report variance across folds.","section":"Tables 4 and 12"}],"minor_comments":[{"comment":"The 'current segment' cosine feature is computed against a segment that contains the target sentence, so it may reflect self-similarity. Please clarify this or remove the feature when claiming a context contribution.","section":"Section 2.2.4"},{"comment":"Please report inter-annotator agreement for the question factuality labels and the answer veracity labels; the current description mentions four annotators but gives no agreement statistics, which would strengthen confidence in the gold standard.","section":"Section 3.1"},{"comment":"The row 'Only context+discourse' has MAP 0.317, which equals Claimbuster-Platform's MAP; the comparison would be clearer if the text noted that this is achieved without the ClaimBuster features.","section":"Table 4"},{"comment":"The neural network hyperparameters (300 epochs, batch size 550) appear to be chosen without an explicit early-stopping or validation-based criterion; please mention whether overfitting was observed or how the hyperparameters were selected.","section":"Section 2.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is plausible but the ablation evidence is confounded in ways that affect the headline conclusion. The inconsistency between the claimed 4.5 MAP improvement and the reported Accuracy metrics in Table 12 is a concrete fixable error. I would encourage the editor to consider the paper after the authors address the confounds and reporting issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. It does real work: new annotations for cQA fact-checking (CQA-QL-FACT), public code and data, a careful comparison against ClaimBuster, and a reasonable multi-task extension. The discourse features and the manual analysis of where they help are a genuine addition over the authors' earlier papers. If you want to know what context-aware features look like for these two tasks, this is a good place to start.\n\nThat said, the stress-test note lands. The central claim that \"context matters most\" is not actually demonstrated by Table 4. The 'Topics' and 'Embeddings' groups each include a 300-dim representation of the target sentence itself, plus three context cosines. When the paper removes 'All\\context', it removes those 600 target-content dimensions along with the position, segment-size, metadata, contradiction, and similarity features. The 4.2 MAP drop could easily come from losing the target sentence's own LDA and embedding signal, not from losing context. This is an internal confound in the experiment the paper relies on. The 'current segment' cosine also includes the target sentence itself, so part of the context score is self-similarity. The cQA ablation in Table 12 is cleaner in that it doesn't include target-content vectors under 'context', but the group bundles thread support, forum-wide evidence, and high-quality-post evidence, so it tests evidence use broadly rather than conversational context. To accept the headline as stated, you'd need a re-run that removes only the context-specific features while keeping the target sentence representation.\n\nOther issues are smaller. There are no error bars or significance tests on the main tables, which matters for a 249-example cQA dataset even if the effect sizes look large. The abstract says the cQA gain is 4.5 MAP points, but Table 12 reports Accuracy; the actual accuracy gain from adding context and discourse is about 0.14. That's a sloppy inconsistency, not a fatal one. The annotation procedure for the cQA data is careful (four annotators, external web verification, explicit exclusion of time-sensitive/ambiguous items), so I don't worry much about label quality.\n\nBottom line: the paper is a legitimate engineering contribution with reusable resources, but the paper's own evidence does not support the \"especially context\" conclusion as cleanly as the text claims. A serious referee should ask for an ablation that separates target-content features from context features, and for significance testing on the small cQA set.\n\nMy recommendation: send it to peer review. With revisions, the contribution stands.","headline":"Solid empirical study, but the headline claim about context is not cleanly isolated by the ablations: the 'context' group bundles target-sentence content (and, in cQA, Web evidence) with genuinely contextual features.","tokens_in":26540,"tokens_out":2266,"would_cite":true,"duration_ms":25071,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatic fact-checking performs best when it models conversational context, not just the wording of the claim.","keywords":["automatic fact-checking","check-worthy claim detection","community question answering","discourse parsing","contextual features","political debates","answer verification","multi-task learning"],"falsifier":"Re-annotate the 249 forum answers with an independent protocol and authoritative sources, then rerun the feature-removal experiments; if removing context no longer produces the reported accuracy drop, the effect was label noise. Alternatively, shuffle or delete the surrounding debate turns while keeping labels fixed and check whether mean average precision still falls by the reported 4.2 points.","tokens_in":25578,"feed_emoji":"🗣️","tokens_out":8771,"duration_ms":80597,"temperature":0.7,"pith_summary":"The paper claims that automatic fact-checking systems are meaningfully better when they model where a claim sits in a conversation, rather than treating the claim sentence in isolation. Across two tasks—deciding which statements in US presidential debates deserve fact-checking, and judging whether forum answers are factually true—context features improve ranking and accuracy more than any other feature group. Discourse structure, such as the rhetorical relations between sentences, also helps, but less than conversational context. If this is right, deployed fact-checkers should draw on surrounding turns, thread-level agreement, and related posts instead of verifying each sentence alone.","feed_headline":"Adding conversational context improves fact-checking the most","feed_subtitle":"Debate turns and forum thread agreement lift performance beyond sentence-only systems.","key_machinery":"The load-bearing mechanism is a feature design that embeds each target sentence in progressively wider rings of context: its position within a speaker turn, the previous and next turns, metadata about speaker and audience reaction, topic and embedding similarity to the surrounding segments, contradiction cues, and, for forum answers, similarity to other answers in the thread and to curated high-quality posts. Discourse context is supplied by a Rhetorical Structure Theory parser, a representation of text as a tree of nucleus–satellite relations, whose features count how the target sentence connects to its neighbors through relations such as Background, Elaboration, and Contrast. The feature-removal experiments—taking each group out and measuring the drop in mean average precision or accuracy—are what carry the argument, since they isolate the contribution of context and discourse.","core_discovery":"The paper establishes that for check-worthiness detection, modeling the target sentence inside its speaker turn and against previous and following turns, speaker metadata, audience reactions, topic distributions, embedding similarity, and contradiction cues yields a ranking system that beats a re-implemented sentence-only baseline by a large margin. Removing the contextual features drops mean average precision from 0.427 to 0.385, while removing the discourse features drops it to 0.412. For answer fact-checking in a community forum, representing an answer against the rest of its thread, against high-quality posts, against the whole forum, and against web evidence reaches 0.683 accuracy; removing context drops that to 0.574 and removing discourse to 0.659. The authors read the consistent pattern across both tasks as evidence that context and discourse carry information that target-sentence features miss.","pith_inferences":["The paper does not develop the point that the context effect may partly be social consensus: forum users tend to repeat correct answers and challenge false ones, so a simpler system based only on repeated-answer agreement might capture much of the same gain.","The discourse priors associated with true statements (Elaboration, Background) are learned from one debate and one forum domain; an obvious extension is to test whether those same relations transfer to news articles, social media, or scientific claims, where discourse conventions differ.","A clean test the paper does not run would hold the gold labels fixed and perturb only the context—shuffling debate turns or removing thread agreement—to quantify how much of the reported gain is genuinely contextual rather than lexical overlap."],"forward_implications":["Fact-checking systems should treat a claim as a node in a conversation, not as an isolated sentence; conversational context is worth roughly 4 MAP points in debate claim ranking and about 11 accuracy points in forum answer verification.","Answer veracity in community forums is learnable separately from answer quality, since a 'good' answer can still be false and thread-level agreement is a usable signal for truthfulness.","Multi-task learning across fact-checking organizations improves the system's ability to mimic any single organization's check-worthiness choices for most of the nine organizations tested, suggesting that joint modeling of editorial policies is useful.","Discourse relations such as Background and Elaboration carry a smaller but non-zero signal, and combining them with context features produces the best overall performance in both tasks."],"supporting_citations":[{"why":"Supplies the CW-USPD-2016 debate dataset with sentence-level check-worthiness annotations from media fact-checkers on which the claim-ranking task and ablations are built.","marker":"[Gencheva et al. 2017]"},{"why":"Provides the previous sentence-level feature set and baseline that the paper re-implements and exceeds for check-worthiness estimation.","marker":"[Hassan et al. 2015]"},{"why":"Provides the CQA-QL-2016 question-answer threads from the community forum that the paper extends with factuality labels.","marker":"[Nakov et al. 2016]"},{"why":"Supplies the Rhetorical Structure Theory discourse parser used to generate discourse relation features in both tasks.","marker":"[Joty et al. 2015]"},{"why":"Supplies the credibility feature set reused for forum answer verification.","marker":"[Castillo et al. 2011]"},{"why":"Supplies the pre-trained word embeddings used for sentence and context similarity features.","marker":"[Mikolov et al. 2013a]"},{"why":"Supplies the textual entailment scores used to compare forum answers against evidence from high-quality posts.","marker":"[Kouylekov and Negri 2010]"},{"why":"Supplies the LDA topic model whose distributions over segments feed the context and topic features.","marker":"[Blei et al. 2003]"}],"fun_headline_variants":["Context beats discourse for automatic fact-checking","Contextual features lift fact-checking more than discourse","Whole-debate context boosts check-worthy claim detection","Thread and forum context drive answer verification accuracy","Context matters: the real driver in fact-checking systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that the gold labels—which media fact-checkers selected and which four annotators judged as true or false using external websites—are correct for every item, so the measured gaps come from context information rather than from label mistakes.","fun_headline_variants_meta":{"raw":{"variants":["Context beats discourse for automatic fact-checking","Contextual features lift fact-checking more than discourse","Whole-debate context boosts check-worthy claim detection","Thread and forum context drive answer verification accuracy","Context matters: the real driver in fact-checking systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000397,"raw_usage":{"total_tokens":2047,"prompt_tokens":885,"completion_tokens":1162,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":1088}},"tokens_in":501,"tokens_out":1162,"duration_ms":12236,"temperature":1.0,"reasoning_tokens":1088,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:15:38.047183+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the 249 forum answers with an independent protocol and authoritative sources, then rerun the feature-removal experiments; if removing context no longer produces the reported accuracy drop, the effect was label noise. Alternatively, shuffle or delete the surrounding debate turns while keeping labels fixed and check whether mean average precision still falls by the reported 4.2 points.","supporting_citations":[],"review_version":1}