{"id":"47a7d8d4-2192-43ad-b672-e732b068988d","arxiv_id":"1908.07491","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Sentence-level context alone is strongly indicative of a concept's controversiality, and simple context-based classifiers match or beat richer metadata-based systems.","lead":"This paper shows that the words surrounding a concept's mentions in Wikipedia sentences can predict how controversial that concept is, using simple classifiers. It also introduces two new datasets, including the first graded 0-10 controversy labels for over 3,500 concepts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim on Dataset I rests on a 2.5-point accuracy gap over 480 concepts with no significance test; the margin may be noise.","rationale":"The reader's verdict is conditional, and I agree with the conditionality, but my main concern differs from the reader's weakest assumption. The ground-truth validity of the Wikipedia controversial-issues list is a real limitation, and the paper itself acknowledges the definition is contested. However, the authors already provide two mitigating controls: leave-one-category-out (§3.3.2) and evaluation on the independently crowd-annotated Dataset III (§4, Table 3). Those controls make the \"context is indicative\" claim reasonably robust, even if the operationalization remains imperfect. The sharper problem is that the SOTA claim, which is the abstract's headline, is supported by a 12-concept margin with no variance estimate. A conditional accept is appropriate, with the condition being a paired significance test and/or confidence intervals for Table 1. If the test shows non-significance, the verdict should move toward rejection of the SOTA claim but not of the underlying context-signal finding. The dataset link placeholder also hurts reproducibility but is secondary to the statistical issue. I therefore set agreement_with_reader to \"partial\": we agree that the paper is conditionally acceptable, but the load-bearing weakness is statistical support, not label validity.","tokens_in":5780,"tokens_out":7199,"duration_ms":76387,"concrete_test":"Obtain the per-concept binary predictions from the 10-fold CV for the context RNN and for the Rad and Barbosa (2012) best classifier on the same 480 Dataset I concepts, run McNemar's paired test on the disagreement cells, and compute a 95% bootstrap confidence interval for the accuracy difference. If p > 0.05 or the interval includes 0, the \"state-of-the-art\" claim should be replaced by \"on par with prior work\"; also report the same interval for the NB-vs-RNN difference on Dataset II.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim has two parts: context is indicative, and simple classifiers are state of the art. The first is supported by large margins over chance and by the leave-one-category-out control. The second is not supported statistically. Table 1 reports context RNN accuracy 0.865 on Dataset I versus 0.84 for the best prior classifier (Rad and Barbosa, 2012), a difference of 0.025, i.e., about 12 concepts out of 480. No confidence intervals, fold-wise variance, or paired significance test are reported. If the difference is within noise, the headline result reduces to \"on par with prior work\" rather than SOTA. The Dataset II results (NB 0.856, RNN 0.841) have no prior comparison at all, so the SOTA claim there is unbenchmarked. The paper's thematic control in §3.3.2 is a good idea, but Table 2 again lacks error bars, so the small drop (RNN 0.865 to 0.856) cannot be interpreted. Because the SOTA claim is the paper's headline contribution, this missing statistical support is the load-bearing weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes predicting the controversiality of Wikipedia concepts from the sentence-level textual contexts in which the concepts are mentioned, with the concept surface form masked. The authors introduce two new datasets: Dataset II, containing 608 controversial concepts from a 2017 Wikipedia list plus matched non-controversial concepts, and Dataset III, containing 3561 edit-protected concepts with crowd-annotated controversiality scores on a 0-10 scale. They evaluate three context-based estimators (a GloVe nearest-neighbor estimator, a naive Bayes classifier, and a bidirectional RNN), and also fine-tune BERT for the Dataset III evaluation. On the existing 480-concept Dataset I, the context-based RNN reaches 0.865 accuracy, compared with 0.84 for the best prior classifier of Rad and Barbosa (2012); on Dataset II, the naive Bayes classifier reaches 0.856. A leave-one-category-out analysis is used to control for thematic leakage. The paper argues that context alone is strongly indicative of controversiality and that simple, language-independent classifiers achieve state-of-the-art results.","tokens_in":6044,"tokens_out":4231,"duration_ms":40481,"significance":"If the results hold, the paper makes a useful contribution: it demonstrates that a lightweight, language-independent signal (immediate sentence context) can compete with complex edit-history and network-based features, and it releases larger labeled datasets, including a graded 0-10 controversiality dataset. The leave-one-category-out control is a sensible and relatively rare sanity check for thematic leakage. The main weakness is statistical: the state-of-the-art claim on Dataset I rests on a 2.5-point accuracy gap over 480 concepts with no confidence intervals or significance tests, and the Dataset II and Dataset III evaluations are likewise reported without variance estimates. The central claim that context is indicative is well supported by large margins over chance, but the more specific claim of state-of-the-art performance is not yet quantitatively substantiated.","major_comments":[{"comment":"The state-of-the-art claim on Dataset I rests on a 2.5-point accuracy gap (0.865 vs. 0.84) over 480 concepts, i.e., roughly 12 correctly classified examples, with no confidence intervals, fold-wise variance, or paired significance test reported. Because this gap is the paper's headline result, the authors should add a paired significance test (e.g., McNemar's test on the 480 concepts) or report bootstrap confidence intervals; without this, the appropriate conclusion is that the context-based RNN is on par with, rather than better than, Rad and Barbosa's best classifier.","section":"§4, Table 1"},{"comment":"The Dataset II results (NB 0.856, RNN 0.841, NN variants around 0.78) are presented as state of the art, but no prior system is evaluated on this dataset, so there is no baseline establishing that these accuracies exceed existing methods. The authors should either run existing methods on Dataset II or explicitly characterize the result as an unbenchmarked evaluation and include a chance-level baseline with significance testing.","section":"§4, Table 1 (Dataset II)"},{"comment":"The Dataset III binary accuracy is computed on one random negative sample of 670 concepts drawn from 1182 concepts with no positive votes, and no variance over random sampling is reported. Since the choice of negative sample can change accuracy nontrivially, the authors should report accuracy over multiple random samples (with mean and standard deviation) or bootstrap confidence intervals.","section":"§3.1, Table 3"},{"comment":"The leave-one-category-out control is a useful sanity check, but the reported accuracies again lack error bars. In particular, the RNN drop from 0.865 to 0.856 on Dataset I is smaller than the likely sampling variability of a 480-concept test set, so the claim that thematic leakage has 'little effect' is not quantitatively supported. Please report fold-wise standard deviations or confidence intervals.","section":"§3.3.2, Table 2"}],"minor_comments":[{"comment":"The dataset download link is given only as the placeholder text 'Datasets'; the URL and access details should be provided.","section":"§3.1"},{"comment":"The RNN section does not specify hidden size, number of layers, dropout, training epochs, or early stopping; enough detail should be added for reproducibility.","section":"§3.2"},{"comment":"The sentence sampling procedure says 'two equal-sized sets' but does not state the number of sentences per concept or the random seed used; please clarify.","section":"§3.1"},{"comment":"The discussion of prior work is brief and would benefit from explicit accuracy figures for each cited method on Dataset I, to make the comparison in Table 1 easier to follow.","section":"§2"},{"comment":"The BERT model is introduced only in the results section; its fine-tuning procedure and hyperparameters should be described in Section 3.2.","section":"§4, Table 3"},{"comment":"The text contains several spacing artifacts from the PDF conversion (e.g., 'Y et', 'T able'); these should be cleaned in the camera-ready version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest about the limitations it does mention, and the datasets are a tangible contribution. The main revision request is statistical support for the state-of-the-art claim; if the authors add significance tests and the gaps shrink, the SOTA wording should be moderated. I see no circularity problem: the labels come from edit-war metadata and crowd annotation, while the features are sentence contexts, which are independent signals. The leave-one-category-out experiment is a good design choice and should be retained."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the dataset, not for the SOTA headline. The 2.5-point accuracy gap on Dataset I (0.865 vs. 0.84) over 480 concepts has no significance test, and the paper's own results section says \"on par\" — the abstract is more assertive than the evidence. That said, the core observation is solid: sentence-level context carries real signal about controversiality, and the leave-one-category-out control shows the signal is not just thematic leakage.\n\nWhat's actually new: a significantly larger graded dataset (Dataset III, 3561 edit-protected concepts with 0-10 crowd scores) and the demonstration that simple, language-independent context features work as well as the complicated edit-history metadata used in prior work. The authors deserve credit for the themed-out control, which explicitly addresses the worry that the model is learning religion- or politics-relatedness rather than controversy per se; the accuracy drop from 0.865 to 0.856 is small. The BERT extension on Dataset III adds another useful data point, and the Pearson correlation of 0.65 for the NB estimator suggests genuine linear relationship with human judgments.\n\nSoft spots, in order of importance. First, the SOTA claim is under-powered. No confidence intervals, fold-wise variances, or paired significance tests are reported anywhere, so the margin over Rad and Barbosa's classifier could be noise. The authors should either add error bars or publicly tone down the abstract to \"on par\" — their own prose already says that. Second, Dataset II has no prior baseline at all, so the \"state-of-the-art\" there is just raw accuracy. Third, the Dataset III binary accuracy relies on a random negative sample of 670 concepts without variance; the correlation scores are the more meaningful metric and should be the headline for that experiment. Fourth, the dataset download link is a placeholder hyperlink, which matters because the data is a major contribution. None of these are load-bearing; they are addressable with standard practice.\n\nWho is this for? Anyone in computational argumentation, controversy detection, or search who wants a better-labeled dataset and evidence that cheap context features can replace expensive metadata. The paper deserves a serious referee and likely acceptance after minor revisions. I would bring it to a reading group and would cite the dataset if I worked in this area.","headline":"The dataset and the context-signal result are worth your time; the SOTA headline is not statistically supported and should be softened.","tokens_in":6533,"tokens_out":2360,"would_cite":true,"duration_ms":25524,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The immediate sentence-level context in which a concept is mentioned is strongly indicative of how controversial it is, and simple context-based classifiers beat existing edit-history methods.","keywords":["controversiality prediction","textual context","Wikipedia","naive Bayes","recurrent neural network","concept classification","dataset","computational argumentation"],"falsifier":"Construct a fresh set of candidate concepts from a source unrelated to Wikipedia's edit-war list—for example, topics argued about in a debate forum or in news comments—have humans label them on the 0-10 scale, and measure whether the context-based models trained on the Wikipedia list agree with those labels at the same accuracy. If the transfer accuracy collapses, the claimed context signal is partly an artifact of the Wikipedia list's biases; a milder test would replace each concept name in held-out sentences with a matched neutral concept and check whether the model's score tracks the identity of the masked concept rather than the surrounding words.","tokens_in":5657,"feed_emoji":"⚖️","tokens_out":5481,"duration_ms":46978,"temperature":0.7,"pith_summary":"The paper claims that how controversial a concept is can be read from the sentences in which the concept appears, and that this textual context alone is enough to match or beat classifiers built on Wikipedia's edit history. Using only the words surrounding a hyperlink to a concept, naive Bayes and a bidirectional RNN reach high accuracy on existing and new binary benchmarks, with the RNN scoring 86.5% on one dataset and naive Bayes scoring 85.6% on a larger new set, without using any metadata or edit-war features. The paper also releases a third, larger dataset of 3,561 edit-protected Wikipedia concepts annotated on a 0-10 controversiality scale, and shows that context-based scores correlate with those human labels. A sympathetic reader would care because it suggests controversiality is a surface-linguistic property detectable from ordinary text, usable in any language or corpus, rather than something requiring costly curation of edit histories.","feed_headline":"Sentence context predicts Wikipedia controversy at 86.5%","feed_subtitle":"Masking the concept name reveals that surrounding words alone rival edit-history classifiers.","key_machinery":"The central object is the sentence-level context of a concept: for every Wikipedia sentence containing a hyperlink to the concept, the hyperlink surface form is replaced by a single fixed mask token, leaving only the surrounding words. The controversy score of a concept is the average, over all such sentences, of a per-sentence score produced either by a naive Bayes bag-of-words model or by a bidirectional RNN with the architecture borrowed from abstractness detection. This masking forces the estimator to judge controversy from context alone, and the averaging over many sentences turns single-sentence signals into a stable concept-level estimate. It is this machinery that makes the approach simple, language-independent, and free of edit-history metadata.","core_discovery":"On the paper's own terms, the central discovery is that the immediate textual context of a concept—the words in Wikipedia sentences that link to it, with the concept name itself masked out—carries strong signal about whether the concept is controversial. The authors demonstrate this by training simple, language-independent classifiers on these contexts and reporting accuracy of 0.865 with the RNN on the first dataset and 0.856 with naive Bayes on the second, comparing favorably with prior classifiers that use collaboration networks and edit-history metadata. In leave-one-category-out tests, where whole thematic categories like Religion or Sexuality are held out from training, accuracy drops only slightly, indicating the models learn something beyond thematic proximity to obviously controversial topics. A further experiment on 3,561 edit-protected concepts with crowd-annotated 0-10 scores shows Pearson correlation around 0.65 between context-based scores and human judgments. The paper concludes that controversiality can be estimated from accessible, abundant textual features rather than complex and sometimes impractical metadata.","pith_inferences":["I infer that the masked-context averaging scheme is not specific to controversy: the same approach could estimate other concept-level properties such as toxicity, sentiment polarity, or political slant with the same language independence.","I infer that controversiality is partly constructed by phrasing: the same concept might receive different scores if named in emotionally loaded versus neutral sentences, which the paper's per-sentence averaging deliberately smooths away.","A testable extension would be to train directly on the 0-10 scale of Dataset III with regression, rather than converting to binary accuracy, and to compare continuous predictions against the full human score distribution.","I infer that the small accuracy drop in leave-one-category-out leaves open the possibility that part of what the models learn is a proxy for topic category rather than controversy itself; the paper acknowledges this, and a stricter test would require matched non-controversial concepts within each category."],"forward_implications":["Controversiality scores can be assigned to any concept in any corpus of moderate size, not just Wikipedia titles, using only the sentences in which the concept is mentioned.","Search engines and content platforms could flag disputed results without accessing an article's edit history, making the signal available for pages on the open web.","The approach supplies an automatic source of debatable topics for argumentation and debate systems, since it can rank concepts by predicted controversiality.","The new continuous-scale dataset allows future work to treat controversiality as a graded quantity rather than a binary label, enabling correlation-based evaluation.","Because the features are plain words, the estimators transfer across languages with little adaptation, as long as sentence-level text is available."],"supporting_citations":[{"why":"Supplies the weakly supervised architecture for scoring concepts from referring sentences, adapted here to controversiality.","marker":"(Rabinovich et al., 2018)"},{"why":"Provides one of the two baselines on Dataset I and the collaboration-network classifiers that the context methods are compared against.","marker":"(Rad and Barbosa, 2012)"},{"why":"Provides the other Dataset I baseline and part of the concept set used as positive examples.","marker":"(Dori-Hacohen et al., 2016)"},{"why":"Supplies the pre-trained GloVe embeddings used by the nearest-neighbor estimator.","marker":"(Pennington et al., 2014)"},{"why":"Provides the BERT model fine-tuned on Dataset I for the correlation experiment on Dataset III.","marker":"(Devlin et al., 2018)"}],"fun_headline_variants":["Context words alone reveal Wikipedia controversy at 86.5%","Masked context predicts controversy without the concept name","Simple language models spot controversy from surrounding words","New dataset scores controversy 0-10, context predicts it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Wikipedia list of controversial issues, curated from edit-war metadata, is a valid and unbiased ground truth for what counts as controversial; if that list is skewed, the supervised models trained on it inherit the skew, and the accuracy numbers are measured against the same list that supplies the training labels.","fun_headline_variants_meta":{"raw":{"variants":["Context words alone reveal Wikipedia controversy at 86.5%","Masked context predicts controversy without the concept name","Simple language models spot controversy from surrounding words","New dataset scores controversy 0-10, context predicts it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1201,"prompt_tokens":841,"completion_tokens":360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":457,"tokens_out":360,"duration_ms":4054,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:17:30.157782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a fresh set of candidate concepts from a source unrelated to Wikipedia's edit-war list—for example, topics argued about in a debate forum or in news comments—have humans label them on the 0-10 scale, and measure whether the context-based models trained on the Wikipedia list agree with those labels at the same accuracy. If the transfer accuracy collapses, the claimed context signal is partly an artifact of the Wikipedia list's biases; a milder test would replace each concept name in held-out sentences with a matched neutral concept and check whether the model's score tracks the identity of the masked concept rather than the surrounding words.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the weakly supervised architecture for scoring concepts from referring sentences, adapted here to controversiality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides one of the two baselines on Dataset I and the collaboration-network classifiers that the context methods are compared against."}],"review_version":1}