{"id":"1e6f9d62-caa1-48a7-a1e7-2d9e906f0dee","arxiv_id":"1908.08147","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"News channels' sentiment on Facebook differs by media type (TV negative, print and radio positive), and users' comment sentiment correlates strongly with post sentiment.","lead":"This paper measures the sentiment of Facebook posts and comments on five news channels, comparing television, radio, and print media. It finds that TV channels post more negative news, print and radio channels post more positive news, and users' comment sentiment tracks post sentiment closely.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Medium-type claim rests on five channels: channel identity and medium are fully confounded, so the 'type of information source' conclusion is not yet supported by the data.","rationale":"The most important claim in the abstract is two-part: post sentiment predicts comment sentiment, and the type of source matters. The first part is supported by per-channel correlations, though these are reported without confidence intervals and the unit of analysis is not explicitly stated. The second part is the paper's distinctive contribution and is structurally underdetermined: the design has five channels, and the medium labels are a fixed property of each channel. No amount of post-level data can separate medium from channel identity without a multilevel model or more channels. I agree with the reader's weakest-assumption analysis; this is the load-bearing limitation. It does not require rejecting the paper: the descriptive findings about these five pages may be correct, and the post-comment correlations are plausible. The large dataset and manual topic precision are real strengths, but they do not mitigate the channel-level confound. The reader's CONDITIONAL verdict is appropriate; I would not change it.","tokens_in":14135,"tokens_out":10836,"duration_ms":117675,"concrete_test":"Run a channel-level permutation test on the existing data: compute the observed mean difference in post sentiment (or mean comment sentiment) between TV and non-TV channels, then randomly reassign the five channels to the three medium labels and recompute the difference. Because there are only 5!/(2!2!1!)=30 distinct assignments, the permutation distribution is exact; if the observed separation is not extreme in that distribution, the medium effect is indistinguishable from channel identity. A complementary check is to fit a multilevel model with posts nested in channels and a random intercept for channel, and report the variance partition coefficient; if channel-level variance dominates, the medium coefficient is not identifiable with five channels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that user-opinion sentiment depends on the 'type of information source' (TV vs print vs radio) is supported only by comparing five Facebook pages: CNN, Fox News, The Economist, NYT, and NPR. Medium groups contain two, two, and one outlets respectively, so the traditional medium of each channel is completely confounded with the channel's identity, editorial policy, audience, and page-specific history. Section 4's observation that the two TV channels look similar and the two print channels look similar is suggestive, but with n=2 per group and n=1 for radio it cannot rule out the alternative that these are just properties of CNN/Fox and Economist/NYT rather than properties of the medium. Table 4's event-level comparison repeats the same five-channel design and inherits the same confound. Because posts within a channel are correlated, the large number of posts does not increase the effective sample size for the medium-level inference; the effective replication count for this claim is five channels. Without a channel-level statistical test or additional outlets per medium, the 'type of information source' component of the abstract's central claim is not identified. The paper may be right descriptively about these five pages, but the general claim about television, radio, and print media goes beyond what the design can support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper examines sentiment in Facebook posts and comments from five news channels (CNN, Fox News, The Economist, NYT, NPR), grouped as television, print, and radio sources. Using VADER for sentiment scoring, LDA topic modeling with manual precision evaluation, and a dataset of 0.15 million posts and 1.13 billion reactions, it reports that TV-based channels post predominantly negative content while print- and radio-based channels post predominantly positive content; that positive and negative posts receive more engagement than neutral ones, with negative posts receiving more comments and shares and positive posts more likes; and that average comment sentiment correlates strongly with post sentiment (0.93-0.98), with channel-level differences. The paper concludes that user opinion sentiment depends on post sentiment and on the type of information source, and it suggests these findings can correct biases when aggregating comments for opinion mining.","tokens_in":14370,"tokens_out":3978,"duration_ms":39344,"significance":"If the findings were generalizable, the paper would provide a useful descriptive account of sentiment dynamics in social media news and practical guidance for opinion-aggregation bias correction. The study's strengths include its unusually large dataset, the use of an externally validated sentiment tool (VADER), manual validation of topic labels (80.3% precision), and a transparent description of the data collection. However, the central claim about the role of the information-source medium is underidentified by the five-channel design, and the correlation analysis lacks the statistical detail needed to support the strong quantitative claims. The qualitative direction of the findings is plausible, but the current support does not yet justify the abstract's general statement about television, radio, and print media.","major_comments":[{"comment":"The claim that post sentiment depends on the type of information source (TV, print, radio) is not identified by the design. The three medium groups contain two (CNN, Fox News), two (The Economist, NYT), and one (NPR) channels, so medium is perfectly confounded with channel identity, editorial policy, and audience. The 150k posts do not increase the effective sample size for the medium-level inference because posts within a channel are correlated; the effective replication count is five. A channel-level analysis or additional outlets per medium is required before the abstract's 'type of information source' claim can be supported. This comment is load-bearing because the medium-type conclusion appears throughout the paper and is repeated in the abstract and conclusion.","section":"Section 4 and Section 4.3 (Figure 2, Table 4)"},{"comment":"The reported correlations between post sentiment and average comment sentiment (0.93-0.98) lack confidence intervals, and the 'p-test' description is insufficient. Because the unit is the post-level average of comment sentiment, within-post comment variance is smoothed away, which can mechanically raise the correlation relative to a comment-level analysis. In addition, post text and comments are scored with the same VADER lexicon, so shared measurement error is a potential source of common signal. The authors should report comment-level or bootstrap/unit-level estimates, provide scatter plots or residual diagnostics, and clarify what exactly was tested with the cited p-value procedure.","section":"Section 6, Table 5"},{"comment":"The popularity-versus-polarity claim (negative posts receive more comments and shares, positive posts more likes) is based on visual comparison of normalized counts without statistical tests or uncertainty estimates. Since the paper contrasts this result with the negativity-bias literature, a regression or bootstrap test with post-level controls (e.g., channel, topic, time) is needed to rule out confounds such as channel-specific posting frequency and topic mix. Without such tests, the claim that the negativity bias operates differently across engagement types is not quantitatively supported.","section":"Section 5, Figures 8-10"}],"minor_comments":[{"comment":"The time-interval column is written as 'Dec 2016-April 2012' for CNN, which reverses chronological order; please use a consistent format such as 'April 2012 - Dec 2016' for all channels.","section":"Table 2"},{"comment":"The text says 'Figures 18-17' when referring to Figure 16 and 17; the figure cross-references need to be corrected.","section":"Section 7"},{"comment":"The conversion of VADER scores from the -1 to +1 range to integers between -5 and +5 is described, but the rounding or binning procedure is not specified; please state it explicitly.","section":"Section 3.2"},{"comment":"The phrase 'p-test' should be 'p-value' or 'significance test'; reference [52] is a cautionary note about p-values, which makes the citation choice puzzling unless the authors intend a specific testing framework.","section":"Section 6"},{"comment":"There are several grammatical slips, such as 'news related to health and world easily catch the attention of the channels' in Section 4.1, where 'news' appears to mean 'news items' and the sentence structure is awkward; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a social-media analytics venue, but the inference about media types requires substantial revision. The dataset availability statement says the data 'will be made available for download from the author's website'; for an empirical paper, the absence of a working link or repository is a concern for reproducibility. On circularity: VADER is an external tool and no constants are fitted to produce the correlations, so I do not see an internal circularity problem. The main risk is overgeneralization from five channels, and the authors should either temper the claims or add channel-level evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper has a genuinely new empirical result: comment sentiment tracks post sentiment across all five channels, with correlations in the 0.93–0.98 range. That directly contradicts Reis et al. [23], who found negative comments posted independently of headline sentiment. Second, the paper's other headline claim — that sentiment strategy depends on the type of information source (TV vs print vs radio) — is not supported by the data as analyzed. The comparison rests on two TV channels (CNN, Fox), two print (Economist, NYT), and one radio (NPR). Channel identity and medium are fully confounded, so the paper can only describe these five pages, not television or print as categories.\n\nWhat the paper does well: the dataset is large (0.15M posts, 1.13B reactions), the VADER choice is sensible for social media text, and the topic-model validation with human judges is a step many papers skip. The engagement finding — that negative posts get more comments and shares but positive posts get more likes — is a clean illustration of negativity bias at different engagement levels, and it is consistent with related work.\n\nWhere it is soft. The medium confound is the main one. With n=2, 2, 1, any claim that 'type of information source' drives the pattern is speculation dressed up as conclusion. The paper even says in Section 4 that the similarity between same-medium channels 'further asserts the influence of the medium' — but that inference does not follow from two examples. Table 4 repeats the same design. This is fixable by either adding outlets per medium or weakening the claim to a case study. The correlations in Table 5 are reported without confidence intervals, scatterplots, or checks for confounding; averaging comments per post tends to inflate correlations, so the true per-comment correlation is probably lower, but the direction is likely right. The p-test reference [52] is not a real statistical test description. Also, the dataset link and the event-selection criteria for Table 4 are missing.\n\nNet: the comment–post correlation is the real contribution and it does not depend on the medium claim. The medium claim is descriptive at best. A serious referee should see this paper; it needs heavy revision but the core observation is worth engaging.\n\nRecommendation: accept for peer review, but with a request to reanalyze the medium comparison at the channel level and report the correlations with diagnostics.","headline":"A useful data-driven paper with a solid comment–post sentiment correlation finding, but the medium-level claim is overreached by a design with five channels.","tokens_in":14865,"tokens_out":3114,"would_cite":true,"duration_ms":29869,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The sentiment of Facebook users' comments tracks the sentiment of the news post and the channel's medium.","keywords":["sentiment analysis","Facebook news pages","news channel comparison","user comments","negativity bias","opinion mining","topic modeling","social media news"],"falsifier":"Re-run the analysis on a much larger panel with many outlets per medium and varied editorial stances. If the correlation between post sentiment and comment sentiment is no stronger within a medium than across media, or if two television pages with opposite editorial slants produce opposite comment reactions, the claim that the medium itself shapes user opinion fails.","tokens_in":13949,"feed_emoji":"📊","tokens_out":7365,"duration_ms":68332,"temperature":0.7,"pith_summary":"This paper claims that, on Facebook news pages, the tone of users' comments is predictable from two things: the sentiment of the news post and the type of channel that posts it. Analysing 0.15 million posts and 1.13 billion reactions from five news pages, it finds that television-based pages post predominantly negative news while print- and radio-based pages post predominantly positive news, and that average comment sentiment follows post sentiment with correlations between 0.93 and 0.98. It also finds that negative posts draw more comments and shares, while positive posts draw more likes, and that the point at which comments turn negative depends on the channel's medium. If these results hold, news outlets' tone choices systematically steer the mood of public discussion, and opinion mining that pools comments across channels inherits a source bias.","feed_headline":"News post tone predicts comment sentiment","feed_subtitle":"On five news pages, comment mood tracks post tone at 0.93–0.98 correlation.","key_machinery":"The analysis is carried by a dataset of 0.15 million Facebook news posts and 1.13 billion reactions from five news pages, combined with VADER, a lexicon-and-rule sentiment scorer specialised for social-media text, which assigns each post and comment a polarity score rescaled to an integer from -5 to +5. Posts are grouped into topics by a probabilistic topic model (latent Dirichlet allocation), and the paper's main quantitative evidence is the per-channel correlation between post polarity and average comment polarity. This machinery turns raw engagement counts into comparable sentiment signals and makes the post-to-comment emotional coupling measurable.","core_discovery":"The paper's central discovery is a strong coupling between the emotional tone of a news post and the emotional tone of the reactions it receives, mediated by the traditional medium of the news source. On the five Facebook pages studied, television news pages are dominated by negative posts, print and radio pages by positive posts, and the same event can be reported with opposite polarity by different types of channels. Average comment sentiment rises and falls with post sentiment for every channel, with correlations from 0.93 for the radio page to 0.98 for one television page. The paper reads this as evidence that users react not only to the news event but to the channel's standing tone: television pages, which mostly post negative news, attract negative comments even for less negative posts, while a public radio page that mostly posts positive news keeps positive average comments regardless of the post's polarity.","pith_inferences":["The medium-level conclusion is entangled with channel identity in the sample: one radio outlet and two each of TV and print leave open the alternative that editorial brand, not the medium, drives the pattern.","The high post–comment correlations may partly reflect commenters quoting or echoing the post's own emotional wording; a follow-up that removes lexical overlap between post and comment text would reveal how much independent opinion remains.","The same-event comparisons hint at a controlled field experiment: post the same factual story with opposite emotional frames on matched pages and measure comment sentiment, which would test the paper's tone-causes-reaction reading against event-driven explanations.","If the coupling holds, a practical extension is an early-warning signal for comment-section toxicity: post tone plus channel tone could flag threads likely to turn negative before many comments arrive."],"forward_implications":["News pages can anticipate the average mood of their comment section from a post's sentiment score before the post is published.","Comment-based opinion mining should adjust for source type; otherwise the dominant tone of a channel leaks into the aggregated public opinion it claims to measure.","The negativity bias needs to be stated at the level of the action: negative posts win comments and shares, positive posts win likes.","Because TV pages skew negative and print/radio pages skew positive, studies comparing audience reactions across media must control for the story, not just the channel, before attributing differences to the medium.","The polarity threshold at which comments turn negative is source-specific, so a single global post-sentiment threshold cannot predict reaction tone across all channels."],"supporting_citations":[{"why":"Supplies the VADER sentiment scorer used to measure polarity of posts and comments.","marker":"[34]"},{"why":"Supplies the Facebook Graph API data-collection path that produced the post-and-comment dataset.","marker":"[32]"},{"why":"Provides the Facebook-reach rationale for choosing Facebook as the platform for the study.","marker":"[3]"},{"why":"Reports that comment polarity is independent of headline sentiment, the prior finding this paper claims to overturn.","marker":"[23]"},{"why":"Provides the significance test used to validate the post-comment sentiment correlations.","marker":"[52]"},{"why":"Documents the short lifetime of negative news, used in the paper's account of why big headlines attract more positive posts.","marker":"[21]"}],"fun_headline_variants":["Post sentiment steers comment sentiment on Facebook","News channel type ties post and comment mood","TV posts negative, print positive: social sentiment split","Reader reactions mirror news post tone","Social news: post emotion predicts comment emotion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that five outlets can stand in for their media types: with one radio page and two each for television and print, the radio-versus-TV differences could be the editorial style of the chosen pages rather than a property of the medium itself.","fun_headline_variants_meta":{"raw":{"variants":["Post sentiment steers comment sentiment on Facebook","News channel type ties post and comment mood","TV posts negative, print positive: social sentiment split","Reader reactions mirror news post tone","Social news: post emotion predicts comment emotion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1392,"prompt_tokens":882,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":444}},"tokens_in":498,"tokens_out":510,"duration_ms":4855,"temperature":1.0,"reasoning_tokens":444,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:47:38.730468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the analysis on a much larger panel with many outlets per medium and varied editorial stances. If the correlation between post sentiment and comment sentiment is no stronger within a medium than across media, or if two television pages with opposite editorial slants produce opposite comment reactions, the claim that the medium itself shapes user opinion fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the VADER sentiment scorer used to measure polarity of posts and comments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Facebook Graph API data-collection path that produced the post-and-comment dataset."},{"cited_title":"Gottfried, E","cited_arxiv_id":null,"evidence_quote":"Provides the Facebook-reach rationale for choosing Facebook as the platform for the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports that comment polarity is independent of headline sentiment, the prior finding this paper claims to overturn."},{"cited_title":"Dahiru, P-value, a true test of statistical signi/f_icance? a cautionary note, Annals of Ibadan postgraduate medicine 6 (1) (2008) 21–26","cited_arxiv_id":null,"evidence_quote":"Provides the significance test used to validate the post-comment sentiment correlations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the short lifetime of negative news, used in the paper's account of why big headlines attract more positive posts."}],"review_version":1}