Pith. sign in

REVIEW 3 major objections 7 minor 10 references

LLM-Supported Content Analysis of Motivated Reasoning on Climate Change

T0 review · 3 major / 7 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read On YouTube climate comment threads, the topic of a comment—government policy or natural cycles versus misinformation—predicts how much reply traffic it gets, with the two settled topics drawing significantly less interaction even after cont

desk verdict The paper's quantitative claim doesn't survive contact with its own outcome variable: the LMER analyzes user-level degree, not comment-level engagement, so the topic coefficients can't show that comments on a topic generate less interaction. read the letter →

arxiv 2508.21305 v1 pith:GGRJCKDQ submitted 2025-08-29 cs.SI

Yuheun Kim , Qiaoyi Liu , Jeff Hemsley This is my paper
classification cs.SI
keywords climatechangemotivatedreasoningYouTubecommentsLLMannotationcontentanalysissocialnetworktopicmodelingpolarization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that in YouTube climate discussions, what a comment is about shapes how much interaction it receives more than whether the video it appears under is pro- or anti-climate-science. Using a large language model to label 44,989 comments into ten topics and coupling those labels with reply-network data, the authors find that comments about government policy and natural cycles draw significantly fewer replies than comments about climate change misinformation. They interpret these quiet topics as ideologically settled points within believer and skeptic communities—statements that affirm group identity rather than invite debate. The finding is offered as evidence of motivated reasoning operating at the conversational level, and the study doubles as a demonstration that LLM-based annotation can make qualitative content analysis tractable at this scale.

What carries the argument

The machinery is a two-stage LLM annotation pipeline combined with a reply-network degree measure. GPT-4o-mini first proposes ten topic categories with written rationales, then labels each of 44,989 comments into those categories; three repeated runs give an average Cohen's κ of 0.84. Each commenter becomes a node in a reply network, and each topic's normalized average degree measures how much interaction comments on that topic receive. A linear mixed-effects model then separates topic effects from video-level effects, with video ID as a random intercept.

What would settle it

Take a random sample of 500 of the 44,989 comments, have two independent human coders assign them to the same ten-topic scheme, compute human–LLM agreement, then rerun the mixed-effects model on the subset where humans and the LLM agree. If the government-policy and natural-cycles effects lose significance, the central claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms: in a linear mixed-effects model with normalized average node degree as the outcome and video ID as a random intercept, comments labeled 'government policy' (β = -0.039, p = .039) and 'natural cycles' (β = -0.038, p = .048) generated significantly lower interaction than the reference topic 'climate change misinformation.' Video type (believer vs. skeptic) was not a significant predictor (β = -0.013, p = .356). The authors conclude that interaction in these spaces is content-driven rather than context-driven, that misinformation functions as a provocation that draws replies, and that policy and natural-cycles comments are echo-chamber statements—ideologically settled

Load-bearing premise

The validity of the results depends on the language model's topic labels matching human-interpretable categories; the paper checks consistency across repeated runs and reviews the labels by hand, but it never compares the model's labels against independent human-coded annotations.

Editorial extensions

If this is right

  • Video stance makes no significant difference to interaction once topic is in the model, so platform-level interventions aimed at one 'side' of the climate debate may miss the more relevant driver: topic-level polarization.
  • Low-interaction topics like government policy and natural cycles are where consensus is reinforced through silence; comments there may be the hardest to dislodge with counter-information.
  • Misinformation-related comments are disproportionately generative of conversation, meaning 'more engagement' is not the same as 'more agreement,' and engagement metrics can inflate the visibility of misinformation.
  • The LLM annotation method, with rationale prompts and cross-run consistency checks, offers a feasible route from tens of thousands of raw comments to interpretable qualitative categories without hiring a coding team.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension: compare the LLM's ten topic labels to independent human coders on a sample and rerun the regression on only the comments where humans and the model agree; this would show whether the significant coefficients are stable under human-interpretable labels.
  • If the pattern generalizes beyond YouTube, communicators should treat issue areas that are 'settled' in a community as needing a different engagement strategy—open questions, novel evidence, or cross-cutting examples—rather than more consensus messaging.
  • The reply-degree outcome counts only comment-reply edges, not likes or views; a broader interaction measure might reveal that policy and natural-cycles comments are consumed widely even if they are not answered.
  • The comments are aggregated at the network level, so an individual-level test—tracking the same users across topics—would clarify whether the same people avoid settled topics or whether distinct subgroups drive the split.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper uses GPT-4o-mini to topic-label a stratified sample of 44,989 YouTube comments from 30 videos (15 pro-ACC "believer" and 15 "hoax" skeptic videos), producing ten topic categories with LLM-generated rationales. It then builds user reply networks and fits a linear mixed-effects model (LMER) with normalized average node degree as the outcome, topic and video type as fixed effects, and video ID as a random intercept. The model reports that comments about 'government policy' (β = -0.039, p = .039) and 'natural cycles' (β = -0.038, p = .048) have significantly lower average degree than the reference topic 'climate change misinformation,' which the authors interpret as evidence that these topics are ideologically settled, low-interaction echo-chamber points reflecting motivated reasoning. The paper also claims video stance (change vs. hoax) does not significantly predict engagement once topic is accounted for.

Significance. If the outcome variable were measuring comment-level reply generation and the LLM topic labels were human-validated, the paper would offer a useful demonstration of LLM-supported qualitative content analysis at scale and a testable claim about how topic content, rather than video stance, shapes engagement in climate YouTube communities. The paper has concrete strengths: the two-step annotation procedure with explicit prompts and rationale outputs is transparent; the inter-run consistency check (Cohen's κ = .84) is reported; the topic table gives interpretable categories; and the network visualizations are an appropriate exploratory tool. However, the central statistical claim rests on an outcome variable that is aggregated at the user/topic level, not at the comment level, and the topic labels lack human-coded validation. Both issues are load-bearing for the interpretation that 'comments on these topics were less likely to generate replies.'

major comments (3)
  1. [Network; Table 2] The model's outcome is 'normalized average node degree' aggregated at the topic-by-video level. Nodes are users, and a user's degree reflects all their reply edges across all their comments, not just the comment(s) assigned to a particular topic. Therefore the coefficients in Table 2 describe the average reply connectivity of users who mentioned a topic, not whether comments about that topic elicited replies. The Results section's wording—'comments on these topics were less likely to generate replies or interaction'—makes the reverse inference. This is a construct-validity problem that remains even if the LLM labels are perfect. Reanalyze at the comment level (e.g., number of replies per comment, with mixed effects for video/thread) or, at minimum, reframe the claim as being about users who post about a topic and justify why that answers the research question.
  2. [Future Directions; LLM Topic Annotation] The paper explicitly acknowledges that it 'lacks the comparison between the model's annotation and human-coded labels.' Cohen's κ = .84 across repeated LLM runs is a reliability measure for the annotator, not a validity measure against human-interpretable categories. The manual review of 500 comments and author re-annotation of 194 outliers are not a systematic human-coding validation. Since all topic assignments used in Table 2 come from this unvalidated pipeline, the central estimates are conditional on the assumption that the LLM's categories match human meaning. Provide a human-coded gold-standard subsample with agreement statistics, or clearly label the analysis as exploratory and LLM-internal.
  3. [Table 2] The headline findings are two nominal p-values (.039, .048) out of nine topic contrasts against the reference category, with no multiple-comparison adjustment. Under a simple Bonferroni correction for nine contrasts, the thresholds would be .0056, and both effects would no longer be significant. Several other contrasts are marginal (.051–.087). Because the paper selects exactly the two significant contrasts and builds the motivated-reasoning interpretation on them, report adjusted p-values or provide a pre-specified hypothesis/correction strategy. Otherwise the central claim is fragile.
minor comments (7)
  1. [Methods: Network] 'Commenter-on pairing' appears to be a typo for 'commenter-to pairing.' Also define what 'fully unnested' means for parent-child reply chains—does it flatten all replies to a single level, and how are nested threads treated?
  2. [Methods: Dataset] The sample is described as 'roughly 10% of the total data,' but 44,989 is about 8.1% of 556,168. Reconcile the number or the claim.
  3. [Table 1] The topic counts in Table 1 sum to 44,798, and adding the 184 'noise' comments gives 44,982, not 44,989. Please reconcile the discrepancy.
  4. [Methods: Network] The term 'normalized average node degree' is not defined. Specify how normalization is performed (e.g., dividing by per-video maximum degree, average degree, or another reference).
  5. [Figure 2] In the small printed figure, node colors and topic labels are difficult to discern. Provide larger panels, a legend, and clearer indication of which nodes are colored by topic.
  6. [Literature Review] There is a typo: 'motivation reasoning' should be 'motivated reasoning' in the first paragraph of the Literature Review.
  7. [Table 2] The significance-code notation is nonstandard (0, ., *, **, ***). Use conventional asterisks and state the corresponding thresholds in the table note.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: topic labels and network degree are independently measured; self-citation is not load-bearing.

full rationale

The paper's analytic chain is empirical rather than derivational: (1) GPT-4o-mini assigns topic labels from comment text alone; (2) reply networks produce normalized average node degree from commenter–commenter edges; (3) an LMER regresses degree on topic and video type with video as a random intercept. The topic variable is not constructed from the outcome, and the outcome is not constructed from the topics. There is no fitted parameter that is renamed as a prediction, no uniqueness theorem, and no equation that defines one result in terms of another. The only self-citation (Liu et al., 2025, supporting 'contrarian scientists' in the literature review) is background and does not bear the weight of any inference. The acknowledged absence of human-coded validation of the LLM labels and the aggregation of degree at the user/topic level are construct-validity concerns, but they are not circularity: even if the labels were wrong, the statistic would still not be true by construction. The 'ideologically settled points' language is an interpretive gloss on low coefficients, not a derived prediction. Therefore score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical regression result. It relies on the assumption that LLM-generated topic labels are valid categories, that average node degree captures engagement, and that the convenience sample represents YouTube climate discourse. No new theoretical entities are introduced.

assumptions (4)
  • domain assumption GPT-4o-mini topic labels are a valid operationalization of qualitative topics.
    The entire regression analysis uses these labels as ground truth categories, but no human-coded validation is performed; only inter-run consistency (Cohen's κ=.84) and author manual review.
  • domain assumption Average node degree in the commenter network is an appropriate measure of user engagement.
    The outcome variable is normalized average node degree, which captures replies within a video; it assumes that more replies indicate more engagement, rather than, say, controversy or conflict.
  • domain assumption The 10% stratified sample is representative of the full comment set.
    The paper states a 10% stratified sample by video type and ID is drawn; this assumes the sample preserves topic and interaction distributions.
  • domain assumption Video selection via top-viewed search results for 'climate change' and 'climate hoax' identifies typical believer/skeptic videos.
    Convenience sampling of top-viewed videos may bias toward viral or mainstream content, but the paper treats them as representative of the two stances.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-Supported Content Analysis of Motivated Reasoning on Climate Change." pith.science (2026). https://pith.science/paper/GGRJCKDQ

@misc{pith2026250821305,
  author       = {Pith},
  title        = {Pith review of: LLM-Supported Content Analysis of Motivated Reasoning on Climate Change},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGRJCKDQ}},
  note         = {Machine review of arXiv:2508.21305}
}
read the original abstract

Public discourse around climate change remains polarized despite scientific consensus on anthropogenic climate change (ACC). This study examines how "believers" and "skeptics" of ACC differ in their YouTube comment discourse. We analyzed 44,989 comments from 30 videos using a large language model (LLM) as a qualitative annotator, identifying ten distinct topics. These annotations were combined with social network analysis to examine engagement patterns. A linear mixed-effects model showed that comments about government policy and natural cycles generated significantly lower interaction compared to misinformation, suggesting these topics are ideologically settled points within communities. These patterns reflect motivated reasoning, where users selectively engage with content that aligns with their identity and beliefs. Our findings highlight the utility of LLMs for large-scale qualitative analysis and highlight how climate discourse is shaped not only by content, but by underlying cognitive and ideological motivations.

Figures

Figures reproduced from arXiv: 2508.21305 by the authors.

Figure 1
Figure 1. Prompts used for Topic Annotation in step 1(left) and step 2(right). [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Social network plots by topics (A and B: change videos with maximum and minimum number of nodes in topic ‘climate change misinformation’. C and D: hoax videos with maximum and minimum number of nodes in topic ‘climate change misinformation’.) A and C are plotted in DRL layout, B and D are plotted in FR layout. Climate change misinformation Climate skepticism Climate solutions Economic impact Environmental activism G… view at source ↗
Figure 3
Figure 3. Average Degree of Nodes by Topic and Video Type [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

10 extracted references · 5 canonical work pages

  1. [1]

    Allgaier, J. (2019). Science and Environmental Communication on YouTube: Strategically Distorted Communications in Online Videos on Climate Change and Climate Engineering. Frontiers in Communication,

    work page 2019

  2. [4]

    https://doi.org/10.3389/fcomm.2019.00036 Al-Rawi, A., OʼKeefe, D., Kane, O., & Bizimana, A.-J. (2021). Twitter’s Fake News Discourses Around Climate Change and Global Warming. Frontiers in Communication,

    arXiv 2019

  3. [6]

    A., Seethi, V

    https://doi.org/10.3389/fcomm.2021.729818 Azher, I. A., Seethi, V. D. R., Akella, A. P., & Alhoori, H. (2024). LimTopic: LLM-based Topic Modeling and Text Summarization for Analyzing Scientific Articles limitations. Proceedings of the 24th ACM/IEEE Joint Conference on Digital Libraries, 1–12. JCDL ’24: 24th ACM/IEEE Joint Conference on Digital Libraries. ...

    work page arXiv 2021

  4. [9]

    online climate change polarization

    https://doi.org/10.3389/fcomm.2024.1301400 Törnberg, P. (2024). Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages. Social Science Computer Review, 08944393241286471. https://doi.org/10.1177/08944393241286471 van Eck, C. W. (2024). Opposing positions, dividing interactions, and hostile a...

    work page arXiv 2024

  5. [14]

    https://doi.org/10.1007/s42001-024-00343-x Zollo, F. (2019). Dealing with digital misinformation: A polarised context of narratives and tribes. EFSA Journal, 17(S1), e170720. https://doi.org/10.2903/j.efsa.2019.e170720

    work page doi:10.1007/s42001-024-00343-x 2019

  6. [24]

    https://doi.org/10.1007/s13278-019-0568-8 Dai, S.-C., Xiong, A., & Ku, L.-W. (2023). LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis (No. arXiv:2310.15100). arXiv. https://doi.org/10.48550/arXiv.2310.15100 de Nadal, L. (2024). From Denial to the Culture Wars: A Study of Climate Misinformation on YouTube. Environmental Communication,...

    arXiv doi:10.48550/arxiv.2310.15100 2023

  7. [35]

    D., Puliga, M., Scala, A., Caldarelli, G., Uzzi, B., & Quattrociocchi, W

    https://doi.org/10.1016/j.cobeha.2021.02.009 Bessi, A., Zollo, F., Vicario, M. D., Puliga, M., Scala, A., Caldarelli, G., Uzzi, B., & Quattrociocchi, W. (2016). Users Polarization on Facebook and Youtube. PLOS ONE, 11(8), e0159641. https://doi.org/10.1371/journal.pone.0159641 Bliuc, A.-M., McGarty, C., Thomas, E. F., Lala, G., Berndsen, M., & Misajon, R. ...

    arXiv 2021

  8. [250]

    https://doi.org/10.1186/s12911-024-02656-3 Mu, Y., Dong, C., Bontcheva, K., & Song, X. (2024). Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, & N. Xue (Eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Re...

    work page doi:10.1186/s12911-024-02656-3 2024

Show all 10 references
  1. [2024]

    10160–10171)

    (pp. 10160–10171). ELRA and ICCL. https://aclanthology.org/2024.lrec-main.887/ Nisbet, M. C. (2009). Communicating Climate Change: Why Frames Matter for Public Engagement. Environment: Science and Policy for Sustainable Development, 51(2), 12–23. https://doi.org/10.3200/ENVT.5...

    2024

  2. [5615]

    https://doi.org/10.1038/s41598-024- 55930-9 Jost, F., Dale, A., & Schwebel, S. (2019). How positive is “change” in climate change? A sentiment analysis. Environmental Science & Policy, 96, 27–36. https://doi.org/10.1016/j.envsci.2019.02.007 Kahan, D. M. (2013). Ideology, motiv...

    2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.