{"id":"9e01c3a4-185f-4043-abb2-cf29c4e3c76b","arxiv_id":"2601.13317","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Paid climate ads on Meta emphasize solution promotion, organic Bluesky posts emphasize critique, and an LLM-augmented clustering pipeline produces themes that outperform LDA/BERTopic in human evaluation.","lead":"The paper compares climate messaging in paid Meta ads with organic Bluesky posts using an LLM-based theme-discovery pipeline, then measures how topics, stances, and responses to political events differ across the two platforms. Paid ads emphasize solution promotion while public posts emphasize critique, and the authors release code and data for the framework.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-platform theme prevalence rests on mismatched sampling frames and dedup asymmetry; Bluesky event-window counts may be sampling noise.","rationale":"The reader's conditional verdict is appropriate: the corpus-construction asymmetry is the most load-bearing risk to the central claim. Agreeing does not mean the claim is false; the descriptive contrast is plausible and the dataset contribution is real. The additional precision here is that the dedup asymmetry (6.7x vs 1.04x) makes Meta counts 'unique creative units' while Bluesky counts are 'nearly raw sampled posts,' so Fig. 3 may be comparing different units. The event-window analysis on Bluesky is particularly fragile because the 20k sample yields small per-window counts. These are correctable with additional computation or by toning claims to 'unique deduplicated texts' with uncertainty estimates. Since the reader already reached CONDITIONAL, no verdict change is needed; a UNCHANGED recommendation preserves that position while specifying the empirical test that would upgrade or downgrade confidence.","tokens_in":20728,"tokens_out":6835,"duration_ms":65305,"concrete_test":"Re-run the comparison using the full ~1.3M Bluesky post set (or a substantially larger random sample, e.g., 200k) with the same keyword filtering and SBERT dedup threshold as Meta, then recompute the Fig. 3 common-theme counts and the Fig. 6 three-day pre/post event-window theme presence, reporting bootstrap 95% confidence intervals for all proportions. If the qualitative prevalence contrasts and the claimed before/after theme emergences persist with non-overlapping intervals, the comparability concern is answered; if they shift or lose significance, the cross-platform prevalence and temporal-responsiveness claims must be restricted or re-analyzed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim (Sec. 6.3, Fig. 3; Sec. 6.4.2, Fig. 6) requires that theme counts measure comparable quantities on Meta and Bluesky. Section 3 constructs them differently: Meta is the complete set of 113,493 keyword-matched ads deduplicated by SBERT cosine >= 0.80 to 17,026; Bluesky is a random sample of 20,000 posts from ~1.3M, deduplicated to 19,182. The dedup rates differ by a factor of ~6.7 (Meta) vs ~1.04 (Bluesky). Because Meta ads are frequently near-duplicate campaign creatives, dedup collapses paid amplification into one representative text, while Bluesky posts are nearly all retained. The prevalence comparison in Fig. 3 therefore compares deduplicated unique ads with near-raw sampled posts, and it is not demonstrated that this is monotonic in actual platform activity (impressions/exposure on Meta vs posting volume on Bluesky). Bluesky counts are also random-sample estimates without confidence intervals; e.g., Climate Wildfires (37 vs 477) and other rare themes could shift under sampling. The event-window analysis compounds this: with ~20k posts sampled over ~15 months, a 3-day window contains only about 130 sampled posts, so 'themes absent before the election' may be sampling zeros rather than sudden activation. Section 6.3's assertion that remaining differences reflect platform incentives rather than labeling artifacts is not tested quantitatively.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a comparative, corpus-level analysis of climate discourse on two structurally different platforms: paid advertisements from the Meta Ad Library (113,493 keyword-matched ads, deduplicated to 17,026) and organic posts from Bluesky (a random sample of 20,000 from ~1.3M keyword-matched posts, deduplicated to 19,182), covering July 2024–September 2025. The authors introduce an interpretable thematic discovery pipeline: SBERT embeddings, PCA/UMAP, HDBSCAN clustering, LLM-based coherence filtering, summarization, cluster merging, and LLM-generated theme labels. Theme assignments are evaluated against LDA and BERTopic using both human and LLM judges, and the themes are further probed via stance prediction and theme-guided retrieval. The main empirical claims are that paid advertising centers on solution-promotion and forward-looking institutional messaging, while Bluesky centers on systemic critique and crisis-oriented discourse; that these differences are reflected in theme prevalence, stance alignment, and temporal responsiveness; and that the induced themes are more coherent and stance-consistent than baseline topic models.","tokens_in":21095,"tokens_out":3398,"duration_ms":34886,"significance":"If the central comparative claims hold, the paper makes a useful contribution to computational social science and NLP: it jointly analyzes paid ads and organic social media discourse, an underexplored comparison, and proposes an interpretable, seed-free thematic discovery pipeline that could generalize to other cross-platform settings. The authors are explicit that the analysis is descriptive rather than causal, and they acknowledge limitations such as LLM bias and the absence of bias mitigation. The release of the dataset and code is a concrete strength, as is the use of human evaluation as the primary signal for theme quality. However, the quantitative support for the headline cross-platform prevalence and event-responsiveness claims is currently weaker than the prose suggests, primarily because the two corpora are constructed under different sampling and deduplication regimes. The paper's descriptive contrasts are plausible, but several load-bearing comparisons need more careful uncertainty quantification and sensitivity analysis before the strongest conclusions are warranted.","major_comments":[{"comment":"The cross-platform prevalence comparison in Fig. 3 rests on non-comparable sampling frames. Meta uses the complete set of 113,493 keyword-matched ads deduplicated to 17,026 (a 6.7x reduction), while Bluesky uses a random sample of 20,000 posts from ~1.3M, deduplicated to 19,182 (a 1.04x reduction). Because Meta ads are often near-duplicate campaign creatives, deduplication collapses paid amplification into one representative text, while Bluesky posts are almost all retained. It is not shown that theme counts after this asymmetric deduplication are monotonic in actual platform activity (impressions/spend for Meta vs. posting volume for Bluesky). Moreover, Bluesky counts are random-sample estimates with no confidence intervals; a theme such as Climate Wildfires (37 vs. 477 in Fig. 3) could shift under sampling. The claim in Sec. 6.3 that 'remaining differences reflect platform-specific inc","section":"Section 3; Section 6.3, Fig. 3"},{"comment":"The event-window analysis for Bluesky is vulnerable to sampling zeros. The Bluesky corpus consists of ~20k randomly sampled posts over ~15 months, so a three-day window contains on average about 130 sampled posts. A theme with even a 1% background prevalence would be expected to appear only ~1.3 times in that window. The statement that four of five themes were 'entirely absent in the three days preceding the election' and that this indicates 'sudden activation of previously latent frames' is not supported without accounting for the sampling rate. Please report raw counts, per-theme rates with Poisson or bootstrap confidence intervals, or conduct the event analysis on the full ~1.3M post corpus instead of the random sample.","section":"Section 6.4.2, Fig. 6"},{"comment":"The stance prediction results are used to validate that themes capture 'stance-relevant signal beyond surface text,' but the evaluation is based on only 500 manually annotated posts per platform, with an 80/20 stratified split yielding a test set of roughly 100 posts. On Bluesky, the stance distribution is extremely imbalanced (4 Pro-Energy posts out of 500), so macro-F1 differences such as 0.46 vs. 0.47 or 0.82 vs. 0.82 are likely within noise. No confidence intervals, significance tests, or variance estimates are reported. This weakens the claim that themes consistently improve stance classification, especially on Bluesky. Please provide uncertainty estimates, consider evaluating on a larger annotation set, or temper the claim accordingly.","section":"Section 7.1, Table 3"},{"comment":"The human evaluation is the primary evidence for theme quality, but the paper does not report inter-annotator agreement (e.g., Cohen's kappa or Krippendorff's alpha). The text states that four researchers annotated in three sessions, but gives no per-annotator breakdown or agreement statistic. Without this, it is difficult to assess whether the reported accuracy differences between methods reflect semantic quality or annotator idiosyncrasy. Please add agreement measures and a more detailed annotation protocol.","section":"App. C.3"}],"minor_comments":[{"comment":"Typo: 'Sustainable Fasion' should be 'Sustainable Fashion'. Also, '8 to-tal' appears to be a typo.","section":"Sec. 6.3"},{"comment":"Typo: 'Principle Component Analysis' should be 'Principal Component Analysis'.","section":"Sec. 5"},{"comment":"The sentence 'our Text-to-Theme method outperforms Text-to-Theme' should presumably read 'outperforms Text-to-Summary'.","section":"App. C.2"},{"comment":"The table header 'ACCF1' is ambiguous; please label columns as Accuracy and Macro F1.","section":"Table 3"},{"comment":"The figure shows top themes but does not indicate whether the bars are counts, proportions, or something else. Please clarify the y-axis and add counts or confidence intervals.","section":"Fig. 6"},{"comment":"For LDA, the number of topics and hyperparameter settings are not reported; please provide these details for reproducibility.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting comparison, and the qualitative examples in Fig. 1 and App. D are compelling. My main concern is that the quantitative comparisons—especially Fig. 3 and Fig. 6—currently rest on sampling and deduplication asymmetries that are not addressed. I would be willing to consider a revised version that adds uncertainty quantification, sensitivity analyses, or a more carefully matched corpus construction. The descriptive narrative may well survive such scrutiny, but the current evidence is not yet proportionate to the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what you should know: this paper is a genuine empirical attempt to compare paid climate advertising on Meta with organic climate posts on Bluesky, and it releases the data and code. The central qualitative finding — paid ads are promotional, solution-oriented, formal; Bluesky posts are critical, crisis-oriented, science-rooted — is plausible and well illustrated. But the quantitative prevalence comparisons and the event-window claims are built on two corpora that aren't comparable, and the paper doesn't acknowledge that.\n\nWhat's new and good: the simultaneous comparison itself appears to be new in the cited literature. The seed-free LLM theme pipeline is a small extension of the authors' earlier machine-in-the-loop work, but it works: themes are more interpretable than LDA or BERTopic under human evaluation, and the human eval is the primary signal, which is the right call given the LLM-judge circularity they themselves note. The stance-prediction and retrieval probes are honest validity checks, and the released resources are useful.\n\nWhere it's soft: Section 3 constructs the corpora differently. Meta is the full 113k keyword-matched ads, deduped by SBERT cosine >= 0.8 down to 17k. Bluesky is a random 20k sample from 1.3M posts, deduped to 19k. The dedup rates differ by a factor of six. So Fig. 3 compares deduplicated unique ad creatives with near-raw sampled posts. That's not a valid basis for claims like 'Clean energy appears more frequently in advertising' or 'Climate Wildfires is more prevalent in organic discourse.' The Bluesky counts are also sample estimates without confidence intervals, so rare-theme gaps like 37 vs 477 could be sampling noise. The event-window analysis has the same problem: a 3-day window around the election contains only a few hundred sampled posts, so 'themes absent before the election' is consistent with sampling zeros.\n\nThe paper asserts remaining differences reflect platform incentives, but that's not tested. Also, no significance tests, no inter-annotator agreement numbers. These are all fixable: sample both platforms the same way, report weighted estimates and CIs, or tone down the claims to qualitative observations. The qualitative contrast itself I find convincing; the numbers just don't yet support the precision the prose claims.\n\nBottom line: this deserves a serious referee, not a desk reject. I'd want major revisions on the sampling-frame issue before relying on any prevalence numbers. If you work on climate discourse or cross-platform methods, read it for the dataset and the framework, not for the current effect sizes.","headline":"Worth engaging for the new Meta–Bluesky dataset and the plausible paid-vs-organic contrast, but the prevalence and event-window numbers rest on mismatched sampling frames and shouldn't be cited without revision.","tokens_in":21548,"tokens_out":3652,"would_cite":true,"duration_ms":39242,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that platform-level incentives—not just topic choice—shape climate discourse online: paid Meta ads concentrate on promoting specific solutions in a formal, forward-looking register, while Bluesky posts concentrate on syste","keywords":["climate discourse","cross-platform analysis","theme discovery","LLM labeling","Meta Ad Library","Bluesky","stance prediction","event analysis"],"falsifier":"Compare theme prevalences using identical keyword lists, matched sampling rates, and a range of deduplication thresholds: if the 'Clean energy' gap between Meta and Bluesky narrows or disappears, or if the post-election contraction of pro-climate ads reverses under any plausible reweighting, the central platform-incentive claim would be undermined.","tokens_in":20605,"feed_emoji":"💬","tokens_out":3876,"duration_ms":35886,"temperature":0.7,"pith_summary":"The paper tries to establish that the thematic structure of climate discourse online is systematically shaped by the incentive structure of the platform where it appears. Using a new seed-free theme discovery pipeline, it compares 17,026 paid Meta ads with 19,182 public Bluesky posts from July 2024 to September 2025. It finds that ads emphasize solution promotion—clean energy, solar, infrastructure—while posts emphasize systemic critique—climate rollbacks, science under attack, capitalism's climate doom. It also shows that pro-climate ad themes contracted sharply after the November 2024 U.S. election, while Bluesky themes surged rapidly in response to political events. The paper argues these differences reflect platform-level incentives, not individual intent, and validates the discovered themes through stance prediction and retrieval tasks.","feed_headline":"Ads sell clean energy, posts warn of climate doom","feed_subtitle":"New theme-mining shows Meta ads push solutions while Bluesky posts push critique—and elections shift ad themes.","key_machinery":"The load-bearing method is a seed-free thematic discovery pipeline: texts are embedded with SentenceBERT, projected with PCA and UMAP, and clustered with HDBSCAN. An LLM (Mistral-Large-Instruct-2407) then filters incoherent clusters, summarizes each cluster, merges redundant clusters based on summary similarity, and assigns concise 1–3 word theme labels. This decouples semantic discovery from theme interpretation, allowing cross-platform comparison without a predefined topic inventory or seed set.","core_discovery":"The central claim is that platform incentives are reflected in the thematic structure, stance alignment, and temporal responsiveness of climate narratives. Paid advertising on Meta centers on strategic promotion of specific solutions in a formal, forward-looking register, while organic posts on Bluesky center on systemic critique in a crisis-oriented, scientifically grounded register. This divergence is systematic across 51 Meta themes and 46 Bluesky themes, with promotion-oriented themes like 'Clean energy' more frequent in ads and critique-oriented themes like 'Anti-Deforestation' and 'Climate Wildfires' more frequent in posts. Temporal analysis shows that after the November 2024 U.S. elec","pith_inferences":["A direct testable extension: apply the same pipeline to other ad libraries (e.g., Google Ads) and other organic platforms (e.g., Mastodon) to see whether the paid/organic thematic divide generalizes or is specific to Meta and Bluesky.","The post-election contraction of pro-climate ad themes may reflect budget reallocation rather than strategic retreat; a causal design that tracks advertiser budgets or matched ad categories could separate these explanations.","The Bluesky stance imbalance (only 4 Pro-Energy out of 500 annotated posts) limits the interpretability of theme-stance correlations there; the paper acknowledges this, but future work could over-sample Pro-Energy posts to stabilize the analysis.","The deduplication threshold (cosine similarity 0.80) could inflate cross-platform prevalence differences; re-running the analysis with multiple thresholds would test the robustness of the 'Clean energy' vs. 'Climate Wildfires' gap."],"forward_implications":["If correct, cross-platform comparisons of climate discourse can be conducted without predefined topic inventories, enabling direct comparison of paid and organic communication across platforms.","Pro-climate ad themes contracted sharply after the November 2024 U.S. election, suggesting that climate advertising volume is tied to electoral cycles and political opportunities.","Bluesky themes like 'Climate rollbacks' and 'Science under attack' surged immediately after political events, indicating rapid activation of latent frames in organic discourse.","The discovered themes improve stance prediction and semantic retrieval performance, suggesting they capture meaningful semantic structure beyond surface-level clusters.","The framework's seed-free design could generalize to comparative thematic analysis in other domains and heterogeneous communication environments."],"fun_headline_variants":["Paid climate ads sell solutions, public posts warn of crisis","Meta ads back clean energy, Bluesky posts back climate alarm","Advertisers pitch climate fixes, organic posts stress urgency","Climate talk splits: ads promote, social feeds critique","Why Meta ads sell green fixes while Bluesky posts spread caution"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the two keyword-matched samples measure the same underlying thing called 'climate discourse,' so that differences in theme counts can be attributed to platform incentives rather than to differences in keyword coverage, sampling rates, or deduplication choices.","fun_headline_variants_meta":{"raw":{"variants":["Paid climate ads sell solutions, public posts warn of crisis","Meta ads back clean energy, Bluesky posts back climate alarm","Advertisers pitch climate fixes, organic posts stress urgency","Climate talk splits: ads promote, social feeds critique","Why Meta ads sell green fixes while Bluesky posts spread caution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":2833,"prompt_tokens":707,"completion_tokens":2126,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":2043}},"tokens_in":451,"tokens_out":2126,"duration_ms":15276,"temperature":1.0,"reasoning_tokens":2043,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T09:33:01.622719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare theme prevalences using identical keyword lists, matched sampling rates, and a range of deduplication thresholds: if the 'Clean energy' gap between Meta and Bluesky narrows or disappears, or if the post-election contraction of pro-climate ads reverses under any plausible reweighting, the central platform-incentive claim would be undermined.","supporting_citations":[],"review_version":1}