Pith. sign in

REVIEW 5 major objections 6 minor 39 references

CommentsRadar: Dive into Unique Data on All Comments on the Web

T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An automated comment-mining pipeline is ready to track news sentiment and find influential commenters.

desk verdict Large-scale comment-mining pipeline with instructive case studies; central suitability claim is plausible but unsupported by any evaluation on actual comment text. read the letter →

arxiv 1908.07069 v1 pith:ES3WRWPL submitted 2019-08-16 cs.IR cs.CLcs.LG

Sergey Nikolenko , Elena Tutubalina , Zulfat Miftahutdinov , Eugene Beloded This is my paper · ORCID
classification cs.IRcs.CLcs.LG
keywords entity-centricsearchonlinecommentsnamedentityrecognitionlinkingsentimentanalysisinfluencerdetectionopinionminingweb
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CommentsRadar, an entity-centric search engine that continuously collects articles and user comments from large news and entertainment sites, extracts and links named entities, measures comment sentiment, and aggregates the results into rankings and timelines. The central claim is that this pipeline, built from current deep-learning tools for named entity recognition, entity linking, and sentiment analysis, is already a suitable tool for finding influential commenters in a media outlet and for tracing how public sentiment toward a news entity changes over time. Three case studies illustrate the claim: a newspaper's readership and influencers, sentiment toward Brexit as the UK began its exit from the EU, and sentiment toward an Instagram celebrity during a 2017 advertising controversy. If the claim holds, journalists, brands, and social media analysts can watch public opinion on named entities at web scale without manual labeling of comment text.

What carries the argument

The load-bearing mechanism is a single pipeline: comment crawlers attach each comment to its article; a pre-trained named entity recognizer extracts entity mentions; a multi-relational neural model with latent relations between mentions assigns each mention to a knowledge-base entry; a convolutional neural network with contextualized word representations classifies comment sentiment; and an aggregation layer turns per-comment scores into per-entity statistics, density rankings, influencer scores, and smoothed sentiment time series. Influencer ranking adapts the bibliometric h-index: a user has index $h$ when $h$ of their comments each received at least $h$ replies, likes, or dislikes. The pipeline is what converts raw comment text into the entity-centered tables and figures the case studies interpret.

What would settle it

Take a random sample of comments from the collected Daily Mail corpus, have annotators label the named entities, their knowledge-base links, and the sentiment, and compare system output against those labels; if entity linking accuracy on the comment text is far below the 70.77% the system achieves on articles, or sentiment agreement falls well below the accuracy reported on standard benchmark data, the conclusion that the pipeline is suitable for comment analysis would be contradicted.

Watch

Extended reading notes

Core claim

The central claim, restated in the paper's conclusion, is that the CommentsRadar approach based on state of the art tools for named entity recognition, named entity linking, and sentiment analysis is already a suitable tool for discovering influencers in media outlets and analyzing sentiment over time for entities that appear in the news. On a corpus of more than 1.67 billion comments attached to over 63 million articles collected through 2017 and early 2018, the pipeline maps mentions to knowledge-base entries, scores each comment with a generic neural sentiment classifier, and aggregates per-entity and per-day statistics. The paper reports that a multi-relational neural entity linking model resolved 70.77% of mentions in a sample of Daily Mail articles, versus 54.14% for an earlier Wikipedia-linking approach, and that the sentiment model's recall on a standard Twitter sentiment benchmark was close to the top-ranked system. In the Brexit case study, smoothed sentiment of Daily Mail comments turned neutral-to-negative beginning in April 2017, in rough agreement with public opinion polls. The paper takes these results as evidence that the end-to-end system is ready for practical use.

Load-bearing premise

The case studies assume that named entity recognition, entity linking, and sentiment models trained or evaluated on news articles, Twitter data, and other benchmark or in-house sets perform about as well on noisy real-world comments, with spelling errors, sarcasm, and colloquialisms; no evaluation on the actual comment text is reported.

Editorial extensions

If this is right

  • A media outlet can rank its commenters by different influence metrics and see distinct populations, with political debate surfacing through comment/reply counts and celebrity-related influence surfacing through like-based h-index scores.
  • Entity sentiment over time can be monitored automatically; the paper's Brexit case shows Daily Mail sentiment moving from neutral to negative in spring 2017, in general agreement with opinion polling.
  • The engine can be offered as a commercial analytics product for advertisers and influencer agencies, identifying which entities and topics a site's audience engages with most.
  • Because frequently discussed entities average toward neutral sentiment, the paper implies that time-series sentiment analysis, rather than pooled averages, is the appropriate way to compare audience feeling across entities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the transfer assumption holds, the same pipeline could be tested as a leading indicator for election polls, brand perception, or market moves; the paper reports only correlation with polls, not predictive power.
  • A direct validation experiment on hand-labeled comments from the indexed sites would settle whether the benchmark NLP results survive contact with real comment noise; the paper does not run it.
  • Because the sentiment model scores entire comments rather than stance toward a specific entity, the system can measure how negative a discussion is but not who the negativity is aimed at; per-entity stance extraction is listed as future work.
  • The influencer results suggest that different media genres should use different influence metrics: volume metrics for political outlets and like-based metrics for celebrity and lifestyle outlets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents CommentsRadar, an entity-centric engine that continuously aggregates articles and user comments from a large set of websites, runs named entity recognition (NER), named entity linking (NEL), and sentiment analysis, and aggregates the results for media analytics and influencer discovery. The authors describe the system pipeline, compare several component models (spaCy and BiLSTM-CRF/ELMo variants for NER; TAGME and a multi-relational neural model for NEL; CNN-based classifiers for sentiment), and report component-level results on standard benchmarks such as OntoNotes 5.0, AIDA-CoNLL, and SemEval 2017. Three case studies are presented: a Daily Mail readership and influencer analysis, a Brexit sentiment time series, and a Kendall Jenner sentiment analysis around a scandal. The conclusion states that CommentsRadar is already a suitable tool for discovering influencers and analyzing sentiment over time for entities in the news.

Significance. If the central claim holds, the paper describes a deployed, large-scale system (1.67 billion comments, 63.1 million articles) with practical value for media analytics, advertising, and social science. The paper gives due credit to standard benchmark evaluations of individual components and contributes a clean adaptation of the h-index to measure commenter influence. The comparison of TAGME with a neural entity-linking model on a domain-relevant corpus and the use of ELMo-enhanced CNNs for sentiment are reasonable engineering contributions. However, the significance of the system-level claim is currently limited because no experiment evaluates NER, NEL, or sentiment analysis on the actual comment text that the system ingests; the case studies provide visual and anecdotal evidence only.

major comments (5)
  1. [Conclusion; Sections 2.2-2.3] The conclusion asserts that CommentsRadar is 'already a suitable tool for discovering influencers in media outlets and analyzing sentiment over time for entities that appear in the news,' but no experiment in the paper evaluates NER, NEL, or sentiment analysis on actual comment text; all component evaluations use OntoNotes newswire/broadcast, AIDA-CoNLL, Daily Mail articles, or SemEval tweets, so the central suitability claim is not supported by the presented evidence.
  2. [Section 2.2, Tables 1 and 2] The text reports that the authors' ELMo-based NER model achieved 76.94% F1 on the Newswire subset of OntoNotes (referencing Table 2), while Table 2 lists 88.34% for the 3-layer BiLSTM(+ELMo)-CRF and Table 1 shows 76.94% as the spaCy model's F1 on Broadcast Conversations; this internal inconsistency undermines the reliability of the component evaluation and must be corrected.
  3. [Section 2.2, NEL evaluation] The entity-linking comparison on the Daily Mail dataset reports only the percentage of entity mentions 'mapped' to Wikipedia pages (54.14% vs 70.77%), which is a coverage rate rather than precision or recall of correct links; without an accuracy evaluation on a manually annotated sample of mentions, the claim that the multi-relational neural model is superior and suitable for CommentsRadar is unsubstantiated.
  4. [Sections 3.2 and 3.3] The sentiment time series for Brexit and Kendall Jenner are interpreted solely by visual alignment with known events and a single YouGov poll, with no quantitative comparison (e.g., correlation or agreement measures) and no manual annotation of comments; the claim that the system reliably analyzes sentiment over time therefore rests on anecdotal validation rather than measured performance.
  5. [Section 2.3 and Section 3] The sentiment model is validated on SemEval 2017 tweets (and in-house data whose results are not reported), but the case-study domains are long-form news comments, which differ in register, length, and noise; without an evaluation on comment-domain text, the sentiment scores used in the case studies are unvalidated.
minor comments (6)
  1. [Author list] The author name 'Mi/f_tahutdinov' is corrupted throughout the paper; the manuscript should be typeset cleanly.
  2. [Table 4] The number of comments for Washington is shown as '1,121,08', which appears to be a typo for 112,108; the reported density of 234.54 is consistent with 112,108/478.
  3. [Section 3.1] The text says 'policial centers' where 'political centers' is intended.
  4. [Section 3.2] The smoothing procedure is described too briefly: the Savitzky-Golay filter parameters and the interpolation method are not specified, making the sentiment curves difficult to reproduce.
  5. [Figure 3] The correlation matrix in Figure 3 is referenced, but its axis labels and color scale are not legible in the provided PDF.
  6. [References] Several web references use inconsistent access-date formatting (e.g., references [6], [17], and [34]); the bibliographic formatting should be harmonized.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NLP components are validated on external benchmarks and case-study outputs are read from the pipeline, not used to fit it.

full rationale

The paper contains no derivation chain in which an output is defined in terms of its own target or in which a fitted parameter is renamed as a prediction. NER is evaluated on OntoNotes subsets (Tables 1 and 2) with models trained on that external corpus; NEL is compared on AIDA-CoNLL and on Daily Mail article text using TAGME and an external multi-relational neural model; sentiment is evaluated on SemEval 2017 Task 4 Subtask A (Table 3) and described as generic. The three case studies report entity densities, sentiment curves, and influencer rankings produced by the pipeline, and the models are not retrained or tuned on the case-study outcome data. No self-citation is load-bearing: references to the authors' own systems or prior work are not used to justify the central suitability claim. The conclusion that CommentsRadar is 'already a suitable tool' is an inductive claim whose support is weakened by the absence of evaluation on actual comment text and by an apparent inconsistency in the reported NER F1 values (the text reports 76.94% on Newswire for the ELMo model while Table 2 lists 88.34% for 3-layer BiLSTM(+ELMo)-CRF and Table 1 lists 76.94% for Broadcast Conversations). These are validity and reproducibility concerns, not circularity: the components are benchmarked against external standards and the case-study conclusions are not fed back into the models. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's central conclusions rest on transfer assumptions: the NLP models are benchmarked on newswire, Wikipedia-linked articles, and tweets, but applied to noisy web comments without end-to-end validation. No free parameters are fitted to the case-study results; the quantitative conclusions, such as 'sentiment became negative', are read directly from model outputs. No invented entities are introduced.

assumptions (4)
  • domain assumption The selected sites and supported comment platforms are representative of public online opinion.
    Used implicitly in Section 2.1 and in the case studies; the paper claims to identify 'trending topics' and 'influencers' based only on these sources.
  • domain assumption Entity mentions in article text can be linked to the same entities discussed in the comments on that article.
    Section 2.2 performs entity linking on article text, then aggregates comments under the same entity. The link between a comment's topic and the article's entity is never validated.
  • domain assumption A single generic sentiment model, trained on tweets and other domains, yields meaningful sentiment scores for comment text across news topics.
    Section 2.3 evaluates on SemEval tweets and 'in-house' data, but applies the model to Daily Mail comments and other sites without a domain adaptation experiment.
  • standard math The Savitzky-Golay smoothing and linear interpolation do not distort the sentiment trends.
    Used in Section 3.2 to create the smoothed red line; this is a standard signal-processing choice, but the smoothing window is not stated, so the trend could be sensitive to it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CommentsRadar: Dive into Unique Data on All Comments on the Web." pith.science (2026). https://pith.science/paper/ES3WRWPL

@misc{pith2026190807069,
  author       = {Pith},
  title        = {Pith review of: CommentsRadar: Dive into Unique Data on All Comments on the Web},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ES3WRWPL}},
  note         = {Machine review of arXiv:1908.07069}
}
read the original abstract

We introduce an entity-centric search engineCommentsRadarthatpairs entity queries with articles and user opinions covering a widerange of topics from top commented sites. The engine aggregatesarticles and comments for these articles, extracts named entities,links them together and with knowledge base entries, performssentiment analysis, and aggregates the results, aiming to mine fortemporal trends and other insights. In this work, we present thegeneral engine, discuss the models used for all steps of this pipeline,and introduce several case studies that discover important insightsfrom online commenting data.

Figures

Figures reproduced from arXiv: 1908.07069 by the authors.

Figure 1
Figure 1. Sentiment evaluation of comments about the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Dataset statistics over time [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Correlation Matrix [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Probability density function (PDF) of the senti [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 7
Figure 7. Figure 7: Sentiment evaluation of Kendall Jenner over time. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 38 canonical work pages

  1. [1]

    2018. In hindsight, do you think Britain was right or wrong to vote to leave the EU? h/t_tps://whatukthinks.org/eu/questions/in-highsight-do-you-think-britain-was- right-or-wrong-to-vote-to-leave-the-eu/?removed=removed&pollster(accessed October 7, 2018)

    work page 2018

  2. [2]

    Amit Agarwal, Ritu Singh, and Durga Toshniwal. 2018. Geospatial sentiment analysis using twi/t_ter data for UK-EU referendum.Journal of Information and Optimization Sciences 39, 1 (2018), 303–317

    work page 2018

  3. [3]

    Christos Baziotis, Nikos Pelekis, and Christos Doulkeridis. 2017. Datastories at semeval-2017 task 4: Deep lstm with a/t_tention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). 747–754. Figure 6: Probability density function (PDF) of the senti- ment score. Figur...

    work page 2017

  4. [4]

    Roi Blanco, Giuseppe O/t_taviano, and Edgar Meij. 2015. Fast and space-efficient en- tity linking for queries. InProceedings of the Eighth ACM International Conference on Web Search and Data Mining . ACM, 179–188

    work page 2015

  5. [5]

    Xilun Chen and Claire Cardie. 2018. Multinomial adversarial networks for multi-domain text classi/f_ication.arXiv preprint arXiv:1802.05694 (2018)

    work page Pith review arXiv 2018

  6. [6]

    Comments Radar [n. d.]. Comments Radar. SolidOpinion. ([n. d.]). h/t_tps: //commentsradar.com/

    work page

  7. [7]

    Marco Cornolti, Paolo Ferragina, and Massimiliano Ciaramita. 2013. A framework for benchmarking entity-annotation systems. (2013), 249–260

    work page 2013

  8. [8]

    Chris Curtis. 2017). /T_here has been a shi/f_t against Brexit, but the public still think we should go ahead with it . h/t_tps://yougov.co.uk/news/2017/10/27/there-has- been-shi/f_t-against-brexit-public-still-t/ (accessed October 7, 2018

    work page 2017

Show all 39 references
  1. [9]

    Paolo Ferragina and Ugo Scaiella. 2010. Tagme: on-the-/f_ly annotation of short text fragments (by wikipedia entities). (2010), 1625–1628

    2010

  2. [10]

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviole/t_te, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks./T_he Journal of Machine Learning Research 17, 1 (2016), 2096–2030

    2016

  3. [11]

    Alec Go, Richa Bhayani, and Lei Huang. 2009. Twi/t_ter sentiment classi/f_ication using distant supervision. CS224N Project Report, Stanford 1, 12 (2009), 2009

    2009

  4. [12]

    Miha Grˇcar, Darko Cherepnalkoski, Igor Mozetiˇc, and Petra Kralj Novak. 2017. Stance and in/f_luence of Twi/t_ter users regarding the Brexit referendum.Compu- tational social networks 4, 1 (2017), 6

    2017

  5. [13]

    Jiafeng Guo, Gu Xu, Xueqi Cheng, and Hang Li. 2009. Named entity recognition in query. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval . ACM, 267–274

    2009

  6. [14]

    Faegheh Hasibi, Krisztian Balog, and Svein Erik Bratsberg. 2016. On the repro- ducibility of the TAGME entity linking system. (2016), 436–449

    2016

  7. [15]

    Jorge E Hirsch. 2005. An index to quantify an individual’s scienti/f_ic research output. Proceedings of the National academy of Sciences 102, 46 (2005), 16569– 16572

    2005

  8. [16]

    Conference’17, July 2017, Washington, DC, USA Nikolenko et al

    Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen F¨urstenau, Man- fred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan /T_hater, and Gerhard Weikum. Conference’17, July 2017, Washington, DC, USA Nikolenko et al

    2017

  9. [17]

    Jigsaw. [n. d.]. Toxic comment classi/f_ication challenge identify and classify toxic online comments. ([n. d.]). h/t_tps://www.kaggle.com/c/ jigsaw-toxic-comment-classi/f_ication-challenge

  10. [18]

    Ossi Karkulahti, Lidia Pivovarova, Mian Du, Jussi Kangasharju, and Roman Yangarber. 2016. Tracking interactions across business news, social media, and stock /f_luctuations. (2016), 735–740

    2016

  11. [19]

    Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classi/f_ication. (2014), 1746–1751

    2014

  12. [20]

    Haewoon Kwak, Changhyun Lee, Hosung Park, and Sue Moon. 2010. What is Twi/t_ter, a social network or a news media? (2010), 591–600

    2010

  13. [21]

    John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random /f_ields: Probabilistic models for segmenting and labeling sequence data. (2001)

    2001

  14. [22]

    Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition. (2016), 260–270

    2016

  15. [23]

    Phong Le and Ivan Titov. 2018. Improving Entity Linking by Modeling Latent Relations between Mentions. arXiv preprint arXiv:1804.10637 (2018)

    2018 arXiv

  16. [24]

    /T_homas Lin, Patrick Pantel, Michael Gamon, Anitha Kannan, and Ariel Fuxman

  17. [25]

    Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial Multi-task Learning for Text Classi/f_ication. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 1–10

    2017

  18. [26]

    Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) . 31–41

    2016

  19. [27]

    Preslav Nakov, Alan Ri/t_ter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoy- anov. 2016. SemEval-2016 task 4: Sentiment analysis in Twi/t_ter. InProceedings of the 10th international workshop on semantic evaluation (semeval-2016) . 1–18

    2016

  20. [28]

    Courtney Napoles, Joel Tetreault, Aasish Pappu, Enrica Rosato, and Brian Proven- zale. 2017. Finding good conversations online: /T_he Yahoo News annotated comments corpus. In Proceedings of the 11th Linguistic Annotation Workshop . 13–23

    2017

  21. [29]

    Ma/t_thew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. 1 (2017), 1756–1765

    2017

  22. [30]

    Ma/t_thew Peters, Mark Neumann, Mohit Iyyer, Ma/t_t Gardner, Christopher Clark, Kenton Lee, and Luke Ze/t_tlemoyer. 2018. Deep Contextualized Word Represen- tations. 1 (2018), 2227–2237

    2018

  23. [31]

    Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twi/t_ter. InProceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) . 502–518

    2017

  24. [32]

    Abraham Savitzky and Marcel JE Golay. 1964. Smoothing and differentiation of data by simpli/f_ied least squares procedures.Analytical chemistry 36, 8 (1964), 1627–1639

    1964

  25. [33]

    spaCy Models Documentation

    spaCy 2017 (accessed October 7, 2018). spaCy Models Documentation. (2017 (accessed October 7, 2018)). h/t_tps://spacy.io/models/en

    2017

  26. [34]

    Statista. 2017. UK: top selling newspapers 2017 — Statista . h/t_tps://www.statista.com/statistics/246077/reach-of-selected-national- newspapers-in-the-uk/ (accessed October 7, 2018)

    2017

  27. [35]

    John Stevens. 2018. Internet Stats & Facts for 2018 . h/t_tps://hostingfacts.com/internet-facts-stats/ (accessed October 7, 2018)

    2018

  28. [36]

    Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Prad- han, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, et al. 2013. Ontonotes release 5.0. Linguistic Data Consortium, Philadelphia, PA (2013)

    2013

  29. [37]

    Sco/t_t Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: /Q_uestion answering with knowledge base. (2015)

    2015

  30. [2011]

    (2011), 782–792

    Robust disambiguation of named entities in text. (2011), 782–792

    2011

  31. [2012]

    In Proceedings of the 21st international conference on World Wide Web

    Active objects: Actions for entity-centric search. In Proceedings of the 21st international conference on World Wide Web. ACM, 589–598

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.