{"id":"d8fb4965-b307-4a5e-8e81-6bd4e0759829","arxiv_id":"1908.07069","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CommentsRadar links named entities to web comments and sentiment, revealing influencer rankings and opinion shifts around Brexit and Kendall Jenner.","lead":"CommentsRadar is an entity-centric search engine that collects articles and user comments from news sites, links them to known people and organizations, and tracks public sentiment about those entities over time. The paper shows three case studies and is a useful reference for building large-scale social listening pipelines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"End-to-end transfer of NER/NEL/sentiment to noisy comment text is unvalidated; the suitability claim is therefore not supported by the presented experiments.","rationale":"The reader's weakest_assumption identifies exactly the vulnerability I see. The central claim is a suitability claim for a deployed pipeline, and such claims require validation on the target data. The paper's component benchmarks come from cleaner or different domains. Since comments include typos, sarcasm, and colloquial language, and since NER/NEL errors propagate into entity-centric aggregation while sentiment errors propagate into time-series analyses, the absence of any labeled-comment evaluation is the load-bearing gap. The internal F1 inconsistency strengthens the need for an independent check of the component numbers. This is not a rejection of the system: the architecture is reasonable, the data scale is large, and the influencer metrics based on interaction counts appear robust. But the conclusion overstates what has been 'confirmed.' The reader's CONDITIONAL verdict is therefore appropriate, and I see no basis to move it.","tokens_in":11305,"tokens_out":3551,"duration_ms":33535,"concrete_test":"Sample about 1,000 comments stratified across the Daily Mail, Brexit, and Kendall Jenner corpora from Section 3; obtain double human annotations for named entity mentions, entity-linking targets, and sentiment polarity. Run the CommentsRadar pipeline (spaCy NER, Le-Titov NEL, ELMo+CNN sentiment) on these comments and compute NER F1, NEL accuracy, and sentiment macro-F1/accuracy. If sentiment macro-F1 on comments is more than about 10 points below the SemEval figure (67.44 F1) or NEL accuracy is substantially below the 70.77% Daily Mail article mapping rate, the transfer assumption fails and the suitability claim should be downgraded. As a minimal secondary check, re-run the NER experiment on OntoNotes Newswire to determine which F1 number (76.94 or 88.34) is correct, since the text and Table 2 currently contradict each other.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The conclusion asserts that CommentsRadar is already suitable for influencer discovery and sentiment-over-time analysis. This requires the NLP components to work on the heterogeneous comment text that the system actually ingests. The paper only validates components on out-of-domain benchmarks: NER on OntoNotes newswire/broadcast (Section 2.2), NEL on AIDA-CoNLL and Daily Mail articles (Section 2.2), and sentiment on SemEval tweets and in-house data (Section 2.3). No experiment measures any component on user comments. The Section 2.2 statement that newswire is 'exactly the type of text we encounter most in CommentsRadar' is asserted, not measured, and comment text is likely noisier due to typos, sarcasm, colloquialisms, and partial sentences. Moreover, the case studies in Section 3 do not provide ground truth: the Brexit and Kendall Jenner sentiment curves are compared only visually with known events and YouGov polls, and entity density tables are produced by the pipeline without error bars. A related internal inconsistency—Section 2.2 text reports 76.94% F1 for the ELMo-based NER model on Newswire while Table 2 lists 88.34% for the 3-layer BiLSTM(+ELMo)-CRF, with 76.94% appearing instead as the Broadcast Conversations F1 in Table 1—shows that even the component validation as reported is not fully reliable. Without an evaluation on actual comments, the central claim is plausible but unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CommentsRadar, an entity-centric engine that continuously aggregates articles and user comments from a large set of websites, runs named entity recognition (NER), named entity linking (NEL), and sentiment analysis, and aggregates the results for media analytics and influencer discovery. The authors describe the system pipeline, compare several component models (spaCy and BiLSTM-CRF/ELMo variants for NER; TAGME and a multi-relational neural model for NEL; CNN-based classifiers for sentiment), and report component-level results on standard benchmarks such as OntoNotes 5.0, AIDA-CoNLL, and SemEval 2017. Three case studies are presented: a Daily Mail readership and influencer analysis, a Brexit sentiment time series, and a Kendall Jenner sentiment analysis around a scandal. The conclusion states that CommentsRadar is already a suitable tool for discovering influencers and analyzing sentiment over time for entities in the news.","tokens_in":11547,"tokens_out":4481,"duration_ms":40907,"significance":"If the central claim holds, the paper describes a deployed, large-scale system (1.67 billion comments, 63.1 million articles) with practical value for media analytics, advertising, and social science. The paper gives due credit to standard benchmark evaluations of individual components and contributes a clean adaptation of the h-index to measure commenter influence. The comparison of TAGME with a neural entity-linking model on a domain-relevant corpus and the use of ELMo-enhanced CNNs for sentiment are reasonable engineering contributions. However, the significance of the system-level claim is currently limited because no experiment evaluates NER, NEL, or sentiment analysis on the actual comment text that the system ingests; the case studies provide visual and anecdotal evidence only.","major_comments":[{"comment":"The conclusion asserts that CommentsRadar is 'already a suitable tool for discovering influencers in media outlets and analyzing sentiment over time for entities that appear in the news,' but no experiment in the paper evaluates NER, NEL, or sentiment analysis on actual comment text; all component evaluations use OntoNotes newswire/broadcast, AIDA-CoNLL, Daily Mail articles, or SemEval tweets, so the central suitability claim is not supported by the presented evidence.","section":"Conclusion; Sections 2.2-2.3"},{"comment":"The text reports that the authors' ELMo-based NER model achieved 76.94% F1 on the Newswire subset of OntoNotes (referencing Table 2), while Table 2 lists 88.34% for the 3-layer BiLSTM(+ELMo)-CRF and Table 1 shows 76.94% as the spaCy model's F1 on Broadcast Conversations; this internal inconsistency undermines the reliability of the component evaluation and must be corrected.","section":"Section 2.2, Tables 1 and 2"},{"comment":"The entity-linking comparison on the Daily Mail dataset reports only the percentage of entity mentions 'mapped' to Wikipedia pages (54.14% vs 70.77%), which is a coverage rate rather than precision or recall of correct links; without an accuracy evaluation on a manually annotated sample of mentions, the claim that the multi-relational neural model is superior and suitable for CommentsRadar is unsubstantiated.","section":"Section 2.2, NEL evaluation"},{"comment":"The sentiment time series for Brexit and Kendall Jenner are interpreted solely by visual alignment with known events and a single YouGov poll, with no quantitative comparison (e.g., correlation or agreement measures) and no manual annotation of comments; the claim that the system reliably analyzes sentiment over time therefore rests on anecdotal validation rather than measured performance.","section":"Sections 3.2 and 3.3"},{"comment":"The sentiment model is validated on SemEval 2017 tweets (and in-house data whose results are not reported), but the case-study domains are long-form news comments, which differ in register, length, and noise; without an evaluation on comment-domain text, the sentiment scores used in the case studies are unvalidated.","section":"Section 2.3 and Section 3"}],"minor_comments":[{"comment":"The author name 'Mi/f_tahutdinov' is corrupted throughout the paper; the manuscript should be typeset cleanly.","section":"Author list"},{"comment":"The number of comments for Washington is shown as '1,121,08', which appears to be a typo for 112,108; the reported density of 234.54 is consistent with 112,108/478.","section":"Table 4"},{"comment":"The text says 'policial centers' where 'political centers' is intended.","section":"Section 3.1"},{"comment":"The smoothing procedure is described too briefly: the Savitzky-Golay filter parameters and the interpolation method are not specified, making the sentiment curves difficult to reproduce.","section":"Section 3.2"},{"comment":"The correlation matrix in Figure 3 is referenced, but its axis labels and color scale are not legible in the provided PDF.","section":"Figure 3"},{"comment":"Several web references use inconsistent access-date formatting (e.g., references [6], [17], and [34]); the bibliographic formatting should be harmonized.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The ACM reference format and copyright notice suggest an original 2016/2017 conference manuscript, while the arXiv posting is dated 2019; the provenance and any prior publication should be clarified. The NER F1 inconsistency and the absence of any evaluation on comment text are the key issues that need to be addressed before archival publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: CommentsRadar is a genuinely large deployed system—1.67B comments from 63.1M articles—and the three case studies are interesting, sometimes sharp (the Kendall Jenner sentiment dip after the Pepsi ad is exactly the kind of insight the tool is for). Second thing: the paper never validates its NLP components on the comment text it actually ingests, so the concluding claim that the approach is 'already a suitable tool' goes beyond what the experiments show. The reader's CONDITIONAL verdict is fair, and the stress-test note is on target.\n\nWhat's genuinely new: the system itself, the dataset scale, and the influencer analysis. The h-index adaptations for commenters—replies, likes, dislikes—are a nice, simple idea, and the observation that like-based and comment-based influence measures diverge is a real, if minor, finding. The component choices are sensible and benchmarked on standard sets, and the paper is honest about building on prior work.\n\nNow the soft spots, roughly in size. Biggest: the transfer gap. NER is tested on OntoNotes newswire/broadcast, NEL on AIDA-CoNLL and Daily Mail articles, sentiment on SemEval tweets and in-house data. None of that is comment text. The paper says newswire is 'exactly the type of text' in CommentsRadar, but that's asserted, not measured, and anyone who has read web comments knows about typos, sarcasm, and fragments. Second: a concrete reporting inconsistency. Section 2.2 says the ELMo-based NER got 76.94% F1 on Newswire, but Table 2 lists 88.34% for the 3-layer BiLSTM(+ELMo)-CRF, and 76.94% is actually the Broadcast Conversations F1 in Table 1. That looks like a copy-paste error, but it undermines confidence in the other numbers. Third: NEL numbers are 'mapped mentions', not precision or recall, so we don't know if the links are correct. Fourth: no code or data release, which matters for a paper whose main contribution is a dataset-plus-system. None of these are fatal for a systems paper, but together they mean the suitability claim isn't yet supported.\n\nWho's this for? Applied NLP and social-computing people who want to see an industry-scale comment-mining pipeline in action. It deserves a serious referee—the scale and case studies justify a review cycle—but the referee should send it back with a demand for a comment-domain evaluation, corrected tables, and at least a demo or data sample. I wouldn't cite it in my own work until those are out.","headline":"Large-scale comment-mining pipeline with instructive case studies; central suitability claim is plausible but unsupported by any evaluation on actual comment text.","tokens_in":12109,"tokens_out":3739,"would_cite":false,"duration_ms":33085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An automated comment-mining pipeline is ready to track news sentiment and find influential commenters.","keywords":["entity-centric search","online comments","named entity recognition","named entity linking","sentiment analysis","influencer detection","opinion mining","web mining"],"falsifier":"Take a random sample of comments from the collected Daily Mail corpus, have annotators label the named entities, their knowledge-base links, and the sentiment, and compare system output against those labels; if entity linking accuracy on the comment text is far below the 70.77% the system achieves on articles, or sentiment agreement falls well below the accuracy reported on standard benchmark data, the conclusion that the pipeline is suitable for comment analysis would be contradicted.","tokens_in":11079,"feed_emoji":"💬","tokens_out":9944,"duration_ms":90643,"temperature":0.7,"pith_summary":"The paper introduces CommentsRadar, an entity-centric search engine that continuously collects articles and user comments from large news and entertainment sites, extracts and links named entities, measures comment sentiment, and aggregates the results into rankings and timelines. The central claim is that this pipeline, built from current deep-learning tools for named entity recognition, entity linking, and sentiment analysis, is already a suitable tool for finding influential commenters in a media outlet and for tracing how public sentiment toward a news entity changes over time. Three case studies illustrate the claim: a newspaper's readership and influencers, sentiment toward Brexit as the UK began its exit from the EU, and sentiment toward an Instagram celebrity during a 2017 advertising controversy. If the claim holds, journalists, brands, and social media analysts can watch public opinion on named entities at web scale without manual labeling of comment text.","feed_headline":"Comment pipeline tracks news sentiment and finds influencers","feed_subtitle":"Brexit comment sentiment tracked opinion polls; rankings split political debaters from celebrity likers.","key_machinery":"The load-bearing mechanism is a single pipeline: comment crawlers attach each comment to its article; a pre-trained named entity recognizer extracts entity mentions; a multi-relational neural model with latent relations between mentions assigns each mention to a knowledge-base entry; a convolutional neural network with contextualized word representations classifies comment sentiment; and an aggregation layer turns per-comment scores into per-entity statistics, density rankings, influencer scores, and smoothed sentiment time series. Influencer ranking adapts the bibliometric h-index: a user has index $h$ when $h$ of their comments each received at least $h$ replies, likes, or dislikes. The pipeline is what converts raw comment text into the entity-centered tables and figures the case studies interpret.","core_discovery":"The central claim, restated in the paper's conclusion, is that the CommentsRadar approach based on state of the art tools for named entity recognition, named entity linking, and sentiment analysis is already a suitable tool for discovering influencers in media outlets and analyzing sentiment over time for entities that appear in the news. On a corpus of more than 1.67 billion comments attached to over 63 million articles collected through 2017 and early 2018, the pipeline maps mentions to knowledge-base entries, scores each comment with a generic neural sentiment classifier, and aggregates per-entity and per-day statistics. The paper reports that a multi-relational neural entity linking model resolved 70.77% of mentions in a sample of Daily Mail articles, versus 54.14% for an earlier Wikipedia-linking approach, and that the sentiment model's recall on a standard Twitter sentiment benchmark was close to the top-ranked system. In the Brexit case study, smoothed sentiment of Daily Mail comments turned neutral-to-negative beginning in April 2017, in rough agreement with public opinion polls. The paper takes these results as evidence that the end-to-end system is ready for practical use.","pith_inferences":["If the transfer assumption holds, the same pipeline could be tested as a leading indicator for election polls, brand perception, or market moves; the paper reports only correlation with polls, not predictive power.","A direct validation experiment on hand-labeled comments from the indexed sites would settle whether the benchmark NLP results survive contact with real comment noise; the paper does not run it.","Because the sentiment model scores entire comments rather than stance toward a specific entity, the system can measure how negative a discussion is but not who the negativity is aimed at; per-entity stance extraction is listed as future work.","The influencer results suggest that different media genres should use different influence metrics: volume metrics for political outlets and like-based metrics for celebrity and lifestyle outlets."],"forward_implications":["A media outlet can rank its commenters by different influence metrics and see distinct populations, with political debate surfacing through comment/reply counts and celebrity-related influence surfacing through like-based h-index scores.","Entity sentiment over time can be monitored automatically; the paper's Brexit case shows Daily Mail sentiment moving from neutral to negative in spring 2017, in general agreement with opinion polling.","The engine can be offered as a commercial analytics product for advertisers and influencer agencies, identifying which entities and topics a site's audience engages with most.","Because frequently discussed entities average toward neutral sentiment, the paper implies that time-series sentiment analysis, rather than pooled averages, is the appropriate way to compare audience feeling across entities."],"supporting_citations":[{"why":"Supplies the multi-relational neural entity linking model used in the pipeline, with 70.77% mention mapping on the Daily Mail sample.","marker":"[23]"},{"why":"Provides the earlier Wikipedia-linking comparison baseline that maps 54.14% of mentions on the same sample.","marker":"[9]"},{"why":"Supplies the reimplementation of that baseline used for the entity linking comparison.","marker":"[14]"},{"why":"Supplies contextualized word representations used in both the entity recognition and sentiment models.","marker":"[30]"},{"why":"Defines the Twitter sentiment benchmark on which the sentiment model's recall is measured and compared with the top-ranked system.","marker":"[31]"},{"why":"Supplies the convolutional neural network architecture for sentence classification used by the sentiment component.","marker":"[19]"},{"why":"Provides the pre-trained named entity recognizer selected for production because its license permits commercial use.","marker":"[33]"},{"why":"Contributes the h-index style influence measures over replies, likes, and dislikes used in the influencer case study.","marker":"[12]"},{"why":"Supplies the smoothing filter applied to daily sentiment scores before interpreting the Brexit and celebrity timelines.","marker":"[32]"}],"fun_headline_variants":["CommentsRadar maps comment sentiment over time and finds influencers","1.67B comments mined for sentiment and influencer trends","Entity-centric engine reveals sentiment swings and key voices","Brexit sentiment tracked from 1.67B comments","CommentsRadar spots influencers from 1.67B comments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The case studies assume that named entity recognition, entity linking, and sentiment models trained or evaluated on news articles, Twitter data, and other benchmark or in-house sets perform about as well on noisy real-world comments, with spelling errors, sarcasm, and colloquialisms; no evaluation on the actual comment text is reported.","fun_headline_variants_meta":{"raw":{"variants":["CommentsRadar maps comment sentiment over time and finds influencers","1.67B comments mined for sentiment and influencer trends","Entity-centric engine reveals sentiment swings and key voices","Brexit sentiment tracked from 1.67B comments","CommentsRadar spots influencers from 1.67B comments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2818,"prompt_tokens":847,"completion_tokens":1971,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1891}},"tokens_in":463,"tokens_out":1971,"duration_ms":13496,"temperature":1.0,"reasoning_tokens":1891,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:59:03.822436+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of comments from the collected Daily Mail corpus, have annotators label the named entities, their knowledge-base links, and the sentiment, and compare system output against those labels; if entity linking accuracy on the comment text is far below the 70.77% the system achieves on articles, or sentiment agreement falls well below the accuracy reported on standard benchmark data, the conclusion that the pipeline is suitable for comment analysis would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the earlier Wikipedia-linking comparison baseline that maps 54.14% of mentions on the same sample."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reimplementation of that baseline used for the entity linking comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies contextualized word representations used in both the entity recognition and sentiment models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Twitter sentiment benchmark on which the sentiment model's recall is measured and compared with the top-ranked system."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the convolutional neural network architecture for sentence classification used by the sentiment component."},{"cited_title":"spaCy Models Documentation","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained named entity recognizer selected for production because its license permits commercial use."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the h-index style influence measures over replies, likes, and dislikes used in the influencer case study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the smoothing filter applied to daily sentiment scores before interpreting the Brexit and celebrity timelines."}],"review_version":1}