{"id":"0136bd0e-c759-4608-9a25-94a9dbc86968","arxiv_id":"2412.16925","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper proposes CSEI, a PCA-weighted composite of sentiment, emotion, readability, offensiveness, and engagement features from 4.5 million Reddit posts, and reports a borderline correlation with 15 COVID-19 events.","lead":"This paper builds a new score, the Community Sentiment and Engagement Index, that combines 13 measures of Reddit discussion about COVID-19 into one daily number. The authors say the score rises and falls with 15 major pandemic events, but their statistical evidence is weak and the validation is mostly in-sample.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The event-validation p=0.0428 is computed from a binary event indicator against an autocorrelated, smoothed CSEI series; the reported significance is not a valid test of event sensitivity.","rationale":"The reader's verdict is REJECT, and my analysis supports that verdict, so no adjustment is needed. I partially agree with the reader's identified weakness: the u_toronto_news dominance is a real representativeness problem, but I see the statistical validity of the event correlation as the more load-bearing flaw. Even if the dataset were perfectly representative, the reported p=0.0428 would not establish event sensitivity because the test ignores autocorrelation induced by the 7-day moving average, uses a sparse binary predictor, and is calibrated on the same data used to build the index and select the peak-alignment narrative. The paper is transparent about its construction: the weights in Eq. 16 sum to 1, the preprocessing pipeline is described, and the use of standard sentiment models is a strength. However, the confirmatory step that carries the central claim is the correlation with events, and that step is not statistically sound. A block-permutation or bootstrap test would settle whether the observed event-day effect is real or an artifact of temporal dependence and sparse events. If that test fails, the central claim is unsupported; if it passes, the dataset-bias concern still warrants caution but the event-response evidence would be materially stronger.","tokens_in":17177,"tokens_out":4143,"duration_ms":40036,"concrete_test":"Compute a null distribution for the event-day effect by block-permuting the 15 event dates (or using a stationary bootstrap with block length 7-30 days) 10,000 times while keeping the CSEI time series fixed, and recompute both the Pearson p-value and the event-vs-non-event mean difference. If the block-permutation p-value exceeds 0.05 or the observed effect falls within the null, the claimed event sensitivity is not supported. As a secondary check, recompute CSEI and the correlation after excluding all posts from u_toronto_news; if the p-value changes materially, the result is dataset-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CSEI responds to major COVID-19 events rests on the Pearson correlation between daily CSEI fluctuations and the 15-event indicator, reported as p=0.0428 in Section IV. This test is statistically invalid as reported. The predictor E_t is binary with only 15 ones across roughly 622 days, and the response is the first difference of CSEI, smoothed with a 7-day moving average (Section III, Eqs. 7-8). A rolling mean induces strong serial dependence in the response; ordinary Pearson p-values assume independent observations, so the effective sample size is far below 622 and the p-value is not trustworthy. In addition, the PCA weights were fit on the same full time series, and the same data were then used to locate peaks, valleys, and prominence thresholds, so the reported alignment of events with peaks is in-sample evidence rather than confirmatory validation. The paper reports no correlation coefficient r, no confidence interval, and no sensitivity analysis for dropping individual events; with only 15 events, one or two influential dates could drive the marginal p=0.0428. The dataset bias flagged by the reader (18.86% of posts from u_toronto_news) is a separate compounding problem, but the statistical flaw alone is sufficient to undermine the event-response claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Community Sentiment and Engagement Index (CSEI), a fixed-weight linear combination of 13 normalized Reddit discourse features (compound sentiment, engagement, readability, offensiveness, fine-grained emotions, domain diversity), with weights derived from the first principal component of a PCA on 4,510,178 COVID-19-related Reddit posts. The authors report that CSEI correlates significantly with its constituent features (all p<0.05), that cumulative CSEI changes show peaks and valleys aligned with 15 selected COVID-19 events, and that a Pearson correlation between daily CSEI changes and a binary event indicator yields p=0.0428. They conclude that CSEI is a sensitive composite barometer of public sentiment and engagement during the pandemic.","tokens_in":17485,"tokens_out":2861,"duration_ms":28129,"significance":"If the claims were properly supported, CSEI would be a useful addition to the toolbox for monitoring public discourse during health crises, and the paper's explicit feature definitions and open weight formula would aid reproducibility. The authors also engage with a very large dataset and apply standard NLP models (VADER, DistilRoBERTa emotion classification, RoBERTa offensiveness, Flesch Reading Ease), which is a strength. However, the paper's central validation is currently in-sample and statistically fragile: the event-response evidence rests on a single borderline p-value from a test whose assumptions are violated, and the 'internal consistency' check is definitionally trivial. The significance of the contribution is therefore not established by the analyses as presented.","major_comments":[{"comment":"The claim that CSEI responds to major COVID-19 events is supported by a Pearson correlation between daily CSEI changes and a binary event indicator E_t, reported only as p=0.0428. This test is statistically invalid as reported. The predictor has 15 ones among roughly 622 days, the response is the first difference of a 7-day moving average (Section III, Eqs. 7–8), which induces strong serial dependence; ordinary Pearson p-values assume independent observations, so the effective sample size is far below 622 and the p-value cannot be trusted. The paper also does not report the correlation coefficient r, a confidence interval, or a sensitivity analysis dropping individual events; with only 15 events, one or two influential dates could drive the marginal p=0.0428. A valid analysis would need to account for autocorrelation (e.g., a permutation or block-bootstrap test) and should report effect size.","section":"Section IV, Eq. (16) and p=0.0428"},{"comment":"The validation is in-sample in two reinforcing ways. First, the PCA weights in Eq. (16) are estimated from the full 4.5M-post dataset, and the same dataset is then used to test event alignment of the resulting CSEI series; there is no train/validation split, temporal holdout, or out-of-sample check. Second, the 15 events, the peak/valley prominence thresholds, and the moving-average window are selected after inspecting the same series, so the reported alignment of peaks and valleys with events is at least partly post hoc. Without an out-of-sample or pre-registered analysis, the event-correlation evidence cannot be interpreted as confirmatory.","section":"Section III–IV, PCA weighting and event validation"},{"comment":"The internal-consistency check—CSEI correlates significantly with each of its constituent features—is definitional. Since CSEI is a weighted average of exactly those 13 normalized features (Eq. 16), any non-degenerate weighting will produce nonzero correlations; high correlations with constituents do not demonstrate that CSEI captures meaningful sentiment and engagement dynamics beyond being a weighted sum of its inputs. This is not evidence of sensitivity to external events. The paper should either present a benchmark against simpler baselines (e.g., an equally weighted average, or a single-feature index) or use external criteria.","section":"Section IV, Figure 1 and surrounding text"},{"comment":"The representativeness of the dataset is a load-bearing assumption for the paper's global claims. The paper itself reports that u_toronto_news contributes 18.86% of posts and autonewspaper another 3.58%, with several other news-bot subreddits (newsbotbot, innews, stardiapostcom, nofeenews) in the top 20. If this composition reflects a scraping or collection artifact rather than the actual distribution of public COVID-19 discourse, then the CSEI time series and the event correlations may largely reflect the posting patterns of a small number of automated or local news accounts. The paper should at least quantify the sensitivity of the CSEI to removal of these dominant subreddits, and should temper claims about 'global' public sentiment accordingly.","section":"Section III, first paragraph and dataset description"},{"comment":"The event-response analysis uses 15 events that are selected post hoc from a known timeline, with no explicit hypothesis about how each event should affect CSEI (direction, magnitude, lag). The peak/valley identification in Eqs. (9)–(13) depends on unspecified distance d and prominence threshold p, and the reported 'alignment' between peaks/valleys and events is not accompanied by any formal matching criterion or statistical test. Given the multiple testing inherent in scanning a 622-day series for 15 event dates, the nominal p=0.0428 does not survive even a mild multiple-comparison correction.","section":"Section IV, event selection and peak/valley analysis"}],"minor_comments":[{"comment":"The displayed equations appear to be missing in the manuscript text (only equation numbers and prose remain); the paper should ensure all equations are rendered, including Eqs. (1)–(15).","section":"Throughout"},{"comment":"The text reports only the p-value for the event correlation; it should report the Pearson r, its confidence interval, and the effective sample size or the method used to account for autocorrelation.","section":"Section IV, p-value reporting"},{"comment":"The figure caption says 'Person's p-values' (typo for Pearson); also, the figure is referenced but not shown in the text, so the reader cannot verify the claimed r-values.","section":"Section IV, Figure 1"},{"comment":"The PCA outlier-removal step uses thresholds ('score below 25 for PC1' and 'at least 7.5 for PC2') that are not justified; it is unclear how these cutoff values were chosen and whether they materially affect the resulting weights.","section":"Section III, outlier removal"},{"comment":"The limitations paragraph acknowledges that the CSEI formula may vary with new data, but this is a general caveat; the paper should also explicitly acknowledge the in-sample validation and statistical issues discussed above.","section":"Section V, limitations"}],"recommendation":"reject","confidential_remarks":"The paper has a clear, reproducible pipeline and a transparent formula, but the central validation is not statistically sound. The p=0.0428 event correlation is computed from an autocorrelated smoothed series against a sparse binary indicator, the PCA weights are fit on the full dataset, and the internal-consistency check is definitional. These are not presentation issues; they undermine the paper's main claim that CSEI is a validated event-responsive index. Even with substantial revision, the current evidence base would need to be rebuilt (proper out-of-sample or permutation testing, sensitivity to dominant subreddits, and a pre-registered or at least clearly non-post-hoc event analysis), which goes beyond a minor revision. I therefore recommend rejection, while encouraging the authors to resubmit a revised version that addresses these statistical and representativeness issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper delivers a specific, clearly specified composite index (Eq. 16) and applies it to a very large Reddit corpus. That part is fine and could be useful. But the validation that the index responds to COVID-19 events is not convincing: the reported p = 0.0428 comes from a test that ignores the autocorrelation induced by the 7-day moving average and the binary event indicator with only 15 ones, and the PCA weights are estimated on the same data used for validation. The internal-consistency check is definitional. So the main claim is unsupported.\n\nWhat is genuinely good: the formula is fully disclosed with all 13 weights, the dataset is large (4.5M posts), and the authors lay out each preprocessing step. The literature review is thorough, and the limitations section is honest.\n\nSoft spots: first, the statistical issue. The Pearson correlation between the smoothed daily change in CSEI and the event dummy is not a valid significance test. With a smoothed, autocorrelated series and 15 event days, the effective sample size is far below 622; p = 0.0428 is meaningless without adjusting for that. Second, the PCA weights are fit on the full time series, and the same series is then used to identify peaks, valleys, and prominence thresholds. That is in-sample confirmation, not validation. Third, the dataset: 18.86% of posts come from u_toronto_news, plus several news-bot subreddits; this distribution likely reflects the scrape, not global public discourse, and the paper does not address it. Fourth, the \"internal consistency\" correlation of CSEI with its own components is true by construction and does not establish sensitivity.\n\nThe good news is that the flaws are in the validation, not in the index construction itself. With a proper out-of-sample or block-bootstrap test, or a time-series model that accounts for autocorrelation, the authors could make the case. As it stands, the paper is a useful description of an index but not a demonstrated measure of event response.\n\nMy recommendation: send it to peer review rather than desk reject. A serious referee can push the authors to redo the validation. If you are short on reviewer budget, a desk reject is defensible, but I would engage.","headline":"A concrete and fully disclosed composite index, but the event-response validation is statistically invalid and in-sample, so the central claim does not hold.","tokens_in":18015,"tokens_out":2844,"would_cite":false,"duration_ms":25112,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A weighted index of 13 Reddit signals tracks public response to 15 COVID-19 events.","keywords":["Community Sentiment and Engagement Index","CSEI","sentiment analysis","Reddit COVID-19 discourse","principal component analysis","fine-grained emotions","engagement metrics","event-driven sentiment shifts"],"falsifier":"Recompute CSEI on the same corpus after removing the subreddit that provides 18.86% of posts and the automated news subreddits; if the $p=0.0428$ event correlation disappears or changes sign, the index is tracking those sources rather than community sentiment.","tokens_in":1642,"feed_emoji":"📊","tokens_out":3825,"duration_ms":90740,"temperature":0.7,"pith_summary":"The paper proposes a single daily number, the Community Sentiment and Engagement Index (CSEI), that summarizes the emotional state and participation level of COVID-19 discussion on Reddit. The index combines 13 measures—overall sentiment, seven fine-grained emotions, posting volume, domain diversity, readability, and offensiveness—into a weighted average, with weights chosen by principal component analysis. The authors claim that when this number is plotted day by day, its peaks, valleys, and cumulative shifts align with 15 major pandemic events, from the WHO pandemic declaration to vaccine approvals, and that the alignment is statistically significant ($p=0.0428$). If right, CSEI gives public-health communicators a near-real-time, text-based barometer of how communities react to crisis events.","feed_headline":"A Reddit index tracks public mood swings across 15 COVID events","feed_subtitle":"The CSEI fuses 13 Reddit discourse signals into one daily number that moves with COVID-19 milestones.","key_machinery":"The carrying mechanism is the PCA-weighted linear formula $\\mathrm{CSEI}(t)=\\sum_i w_i X_i^{\\mathrm{norm}}(t)$, where each $X_i$ is a daily feature computed from Reddit posts and each $w_i$ is the loading of that feature on the first principal component. The 13 features are compound sentiment, daily post count, daily total score, readability, offensiveness, domain diversity, and seven fine-grained emotion shares (fear, surprise, joy, sadness, anger, disgust, neutral). Features are min-max normalized before weighting, which keeps high-count variables from swamping subtle emotion signals while still letting domain diversity and post volume contribute strongly. Peaks and valleys are detected on a 7-day moving average of the daily differences, and an event indicator feeds the Pearson test that links CSEI movement to the 15 pandemic events.","core_discovery":"The paper's claim is that CSEI, a fixed-weight linear combination of 13 normalized Reddit discourse features, captures both the direction and intensity of public response to COVID-19 milestones. The weights come from the loadings of the first principal component, so features that explain more variance in the data carry more weight; the reported equation is Eq. 16, with domain diversity highest at 0.1761, followed by compound sentiment at 0.1398 and offensiveness at 0.1386. The authors show that each constituent feature correlates significantly with the index ($p<0.05$), and that a 7-day rolling mean of CSEI exhibits peaks and valleys whose dates line up with the 15 selected events. They further report a Pearson correlation between daily CSEI changes and an event-day indicator at $p=0.0428$, interpreting this as evidence that the index can infer and interpret shifts in public sentiment and engagement around major events.","pith_inferences":["The validation uses only 15 event indicators spread over 20 months, so the Pearson test has low statistical power; a stronger test would shuffle event dates and compare the observed correlation against that null distribution.","Because the corpus is dominated by a single subreddit and several automated news feeds, the index may partly measure news-bot posting volume rather than human sentiment; reweighting the corpus by subreddit would test this.","An increase in CSEI is not automatically an improvement in community well-being, since offensiveness and high engagement enter positively; readers should interpret CSEI as discourse intensity plus expressed emotion rather than as a health score."],"forward_implications":["A daily CSEI time series can serve as a monitoring tool: a sharp rise after an announcement signals elevated public anxiety or attention, and a valley signals that attention is returning to baseline.","Because the weights are fixed while the feature values move with new posts, the index can be re-run on fresh days of the same crisis to score later events against the same scale.","CSEI extends sentiment analysis beyond positive/negative polarity by treating engagement breadth, civility, and fine-grained emotions as first-class components of community response.","The event correlation at $p=0.0428$ suggests the index is sensitive enough to distinguish event days from ordinary days, which is the property needed for real-time crisis communication.","The methodology can be transferred to other public-health crises by recomputing the feature set and PCA weights on a new Reddit corpus."],"supporting_citations":[{"why":"Supplies the 4,510,178 Reddit posts used to build and validate CSEI.","marker":"[33]"},{"why":"PCA is the weighting method: loadings on the first principal component become the coefficients in Eq. 16.","marker":"[40]"},{"why":"VADER produces the compound sentiment feature in the index.","marker":"[66]"},{"why":"The emotion classifier produces the seven fine-grained sentiment features (fear, surprise, joy, sadness, anger, disgust, neutral).","marker":"[61]"},{"why":"The offensive-language model produces the offensiveness feature in the index.","marker":"[62]"},{"why":"Pearson's correlation is the statistical test used to claim CSEI daily changes relate to the 15 events.","marker":"[42]"},{"why":"Provides the timeline of 15 major COVID-19 events against which CSEI peaks, valleys, and cumulative changes are aligned.","marker":"[74]"}],"fun_headline_variants":["A new index maps Reddit sentiment to COVID milestones","CSEI: a daily Reddit mood gauge for COVID events","Reddit posts power a sensitive COVID sentiment index","Index links Reddit chatter to major COVID moments","Tracking public mood on COVID via a Reddit-derived index"],"cache_read_input_tokens":20096,"weakest_assumption_plain":"The load-bearing assumption is that the 4.5 million Reddit posts—more than a fifth of them from a single news subreddit and several automated accounts—represent what the public actually felt and engaged with, rather than the posting habits of a few bots and one community.","fun_headline_variants_meta":{"raw":{"variants":["A new index maps Reddit sentiment to COVID milestones","CSEI: a daily Reddit mood gauge for COVID events","Reddit posts power a sensitive COVID sentiment index","Index links Reddit chatter to major COVID moments","Tracking public mood on COVID via a Reddit-derived index"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1382,"prompt_tokens":1046,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":272}},"tokens_in":662,"tokens_out":336,"duration_ms":3399,"temperature":1.0,"reasoning_tokens":272,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:58:42.208133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute CSEI on the same corpus after removing the subreddit that provides 18.86% of posts and the automated news subreddits; if the $p=0.0428$ event correlation disappears or changes sign, the index is tracking those sources rather than community sentiment.","supporting_citations":[{"cited_title":"The Reddit COVID dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the 4,510,178 Reddit posts used to build and validate CSEI."},{"cited_title":"A tutorial on principal component analysis,","cited_arxiv_id":null,"evidence_quote":"PCA is the weighting method: loadings on the first principal component become the coefficients in Eq. 16."},{"cited_title":"[Accessed: 10-Nov-2024]","cited_arxiv_id":null,"evidence_quote":"The emotion classifier produces the seven fine-grained sentiment features (fear, surprise, joy, sadness, anger, disgust, neutral)."},{"cited_title":"[Accessed: 10-Nov-2024]","cited_arxiv_id":null,"evidence_quote":"The offensive-language model produces the offensiveness feature in the index."},{"cited_title":"CDC museum COVID -19 timeline,","cited_arxiv_id":null,"evidence_quote":"Provides the timeline of 15 major COVID-19 events against which CSEI peaks, valleys, and cumulative changes are aligned."}],"review_version":1}