REVIEW 3 major objections 5 minor 43 references
Does online sustainability communication shape public discourse? Insights from six years of tenant-housing provider interactions
T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Tenant replies to sustainability posts are topic-aligned but their discourse type is set by provider context, not post design.
desk verdict Interesting framework, but the main regression's unit-of-analysis gap makes the headline organizational-effects result unverifiable as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Each comment is reduced to a seven-feature vector: the probabilities of four communicative intents (appreciation, criticism, inquiry, statement), a binary indicator for forwarding or tagging, a continuous sentiment score, and the cosine similarity between the post's and the comment's sentence embeddings (the 'semantic relatedness' dimension, i.e., how closely the comment stays on the post's subject). K-means clustering of these standardised vectors produces the six discourse types, and a multinomial logistic regression with content sharing as the reference category links those types to post design and organisational covariates.
What would settle it
Take a random sample of roughly 200 comments, have independent human annotators assign sentiment, communicative intent, and on-topic relatedness, then re-run the clustering and the multinomial regression using only those human labels. If the six types do not reappear, or if the organisational predictors lose their association, the central claim would be refuted. A simpler check: re-run the clustering over many random seeds; if the six-cluster structure is not stable, the typology is not a stable feature of the data.
Extended reading notes
Core claim
The paper shows that tenant responses to sustainability communication are structured, multidimensional, and shaped primarily by organisational context rather than message format. Using 792 posts and 3,197 comments, the authors identify six stable discourse types—on-topic feedback, on-topic criticism, on-topic praise, off-topic complaints, content sharing, and information seeking. They also show, via a permutation test, that comments are significantly more similar to their own post than to randomly paired posts, indicating that what an organisation publishes anchors the topic of the responses. In a combined multinomial regression, post-level design features (length, lexical diversity, questio
Load-bearing premise
The load-bearing premise is that the three machine-estimated measures—semantic similarity, intent labels, and sentiment—are accurate enough to reflect what tenants actually do; if any of them misclassify systematically, the six discourse types and the organisational findings could be artefacts of the classifiers rather than real tenant behaviour.
Editorial extensions
If this is right
- Organisations retain agenda-setting capacity in open digital spaces: the semantic-alignment result implies that the topics an organisation chooses to post about steer the subject matter of the comments that follow.
- Rewriting posts—changing length, adding questions, or including links—should not be expected to change the discourse-type mix much; organisational scale, rent levels, and tenant satisfaction carry more predictive weight.
- Simple engagement counts (likes, comment volume) are unreliable proxies for response quality, because equal volumes can mask very different mixes of feedback, criticism, praise, complaints, sharing, and information seeking.
- The rising share of on-topic criticism and information seeking after 2020 signals a qualitative shift in how tenants engage, not merely more activity.
- The measurement pipeline is transferable: the same three-dimensional classification and clustering approach could be applied to other organisations and policy domains as a discourse-quality monitor.
Reading between the lines
- An implication the paper leaves implicit: if post design does not move the discourse mix, an organisation wanting more substantive or less complaint-heavy engagement would be better advised to address structural conditions (rent policy, service quality, scale) than to invest in message styling.
- The machine labels are the leverage point: because intent models were trained on 13,000 synthetic comments seeded from 246 human examples, and individual sentiment models scored between 0.34 and 0.70 accuracy before ensembling, the six clusters and the organisational effects could be partial artefacts of systematic misclassification. A human re-coding of a random comment sample would settle this.
- A testable extension is to classify the organisations' own posts by communicative function—inform, ask, justify, promote—and test whether post function interacts with organisational context to predict discourse types better than the purely formal post features used here.
- The negative association between tenant satisfaction and expressive discourse could reflect satisfied tenants having little to complain about, but it could also proxy for tenant composition (income, age, digital literacy), which the dataset does not measure.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a data-driven framework to characterize tenant comments on Dutch housing associations' Facebook posts along three dimensions—semantic relatedness (embedding cosine), communicative intent (fine-tuned transformer ensemble trained with LLM-generated synthetic data), and sentiment (model ensemble). Comments are clustered via k-means into six discourse types (on-topic feedback, on-topic criticism, on-topic praise, off-topic complaints, content sharing, information seeking). A multinomial logistic regression then relates post-design features (length, lexical diversity, questions, URLs) and organizational characteristics (scale, affordability, tenant satisfaction) to the discourse type, using 648 posts from the top 50 organizations. The paper reports that organizational characteristics dominate, while post-design features are not significant, and that discourse shifts toward criticism and information seeking over time.
Significance. The paper addresses a real gap: moving beyond engagement counts to qualitative discourse structure. The pipeline is clearly specified, and the code availability statement is a strength. The permutation test for semantic alignment is a reasonable empirical check, and the intent/sentiment ensembles are evaluated on held-out human labels. If the central regression result is valid, the finding that organizational context matters more than post wording would be a useful contribution to government social-media research. However, the interpretation of the regression depends on an unstated aggregation step, which (until clarified) makes the headline conclusion unverifiable.
major comments (3)
- [§4.2 and Table G1] The regression's unit of analysis is ambiguous. Discourse types are assigned per comment in §4.1.4, yet Table G1 reports N=648 (posts) while the text mentions 2,882 comments. The manuscript never specifies how comment-level cluster labels are aggregated to a single post-level outcome. Different rules (majority vote, first comment, most frequent label, or random draw) can produce different outcome labels and therefore different coefficients and conclusions. For example, the reported affordability coefficient for on-topic criticism (β=−1.22) and scale coefficient (β=1.08) are uninterpretable without this rule. This is load-bearing because the paper's central claim that organizational characteristics, not post design, shape discourse rests entirely on this regression. The authors must state the aggregation rule, justify it, and ideally re-estimate with a multilevel model or comment-level an
- [§5.2 and Table G1] The claim that 'organisational characteristics systematically dominated the prediction' overstates the evidence. The full model has pseudo R²=0.044, meaning over 95% of the variation in discourse type is unexplained. While some coefficients are statistically significant, their explanatory power is small. To support 'domination', the paper should compare the pseudo R² of a model with organizational characteristics only, a post-features-only model, and the combined model, and should temper the language to acknowledge that the effects are modest in magnitude.
- [§4.2 and Table 2] Restricting the regression to the top 50 organizations by comment count introduces selection on the dependent variable. Organizations with more comments are overrepresented, which may bias the estimated associations between organizational characteristics and discourse type. At minimum, the authors should discuss this truncation as a limitation and report a robustness check using the full set of 92 organizations (for example, a random-effects model) if possible.
minor comments (5)
- [Table G1] The label 'On-topic Feedbacks' is inconsistent with the text's 'on-topic feedback' and is ungrammatical; please unify terminology.
- [§4.1.2] The intent classifier's per-class performance is weak for Appreciation (F1=0.67, support=7). This small support should be flagged when interpreting cluster composition and downstream regression results for praise-related discourse types.
- [§4.1.4] The choice of k=6 is based only on the silhouette score. A stability check (e.g., bootstrapped clustering) would strengthen the claim that the six discourse types are stable; currently this is a potential source of arbitrariness.
- [§7] The code availability statement says 'The code is available here' but no URL is provided. Please include a link or repository identifier.
- [Appendix E] Table E1 lists individual sentiment model accuracies, but several models (e.g., DTAI-KULeuven/robbert-v2-dutch-sentiment at 0.42) are below random for a 3-class problem; this reinforces the need to report per-class metrics for the chosen ensemble beyond overall accuracy.
Circularity Check
Minor self-definitional validation in §5.1; the substantive organizational-effects claim rests on independent regression and permutation evidence.
-
self definitional
[Section 5.1, Discourse Typology (paragraph after Figure 2)]
"These results demonstrate that discourse types emerge as structured configurations of the three dimensions ... More broadly, these findings validate the multidimensional operation of discourse: the identified clusters arise directly from the joint distribution of semantic relatedness, sentiment, and communicative intent, confirming that these dimensions capture meaningful and recurrent patterns of citizen engagement."
The six clusters are constructed by k-means on exactly the seven features described in §4.1.4: five intent probabilities, a sentiment score, and a relatedness score. The sentence treats the resulting cluster separation as confirmation that the three dimensions 'capture meaningful and recurrent patterns.' But k-means is optimized to separate points on those input dimensions, so cluster-level alignment with the input features is guaranteed by the clustering objective rather than being independent evidence for the meaningfulness of the dimensions. This is a self-definitional validation step, though it is not the basis of the later organizational-effects finding, which uses external Aedes-benchmark predictors and a permutation test.
full rationale
The paper's central empirical claims are largely self-contained and do not reduce to their inputs by construction. The permutation test in §4.1.1 compares real post-comment embedding similarities with randomly paired content; this is an independent empirical comparison, and the result could in principle have gone either way. The multinomial logistic regression in §5.2 uses organizational features (housing stock, rent, satisfaction) obtained from external Aedes-benchmark data, not from the discourse labels, so the headline organizational-effects results are not fitted from the outcome. The only self-citation (Ossokina et al. 2021) supports a contextual legal claim about tenant approval and is not load-bearing for the discourse typology or regression. The skeptic's concern about unspecified aggregation of comment-level clusters to post-level outcomes is a reproducibility/validity issue, not circularity: it does not entail that the outcome was constructed from the predictors. One minor circularity exists in §5.1, where clustering on the three dimensions is presented as validating those dimensions; this is rhetorical self-confirmation but does not affect the independent organizational-effects analysis. Overall, the derivation chain is not circular in its main claims; the score reflects only this minor self-definitional validation.
Assumptions & free parameters
free parameters (4)
- Number of discourse clusters k =
6
- Binarization thresholds for organizational and post features =
33rd/67th percentiles
- Synthetic intent training data size =
13,000 examples
- k-means random seed =
42
assumptions (5)
- domain assumption Cosine similarity of paraphrase-multilingual-MiniLM-L12-v2 embeddings is a valid measure of semantic relatedness for short Dutch Facebook comments.
- domain assumption LLM-generated synthetic comments preserve the distribution and style of real tenant Facebook comments well enough to train intent classifiers.
- domain assumption The five-way intent taxonomy and three-way sentiment scheme capture the meaningful dimensions of citizen discourse.
- domain assumption k-means with standardized seven-dimensional features yields clusters that correspond to interpretable discourse types.
- domain assumption The rule-based sustainability filter and the requirement of at least one comment identify organizational sustainability communication.
Cite this review
Pith. "Pith review of Does online sustainability communication shape public discourse? Insights from six years of tenant-housing provider interactions." pith.science (2026). https://pith.science/paper/NX6RCZBE
@misc{pith2026260708437,
author = {Pith},
title = {Pith review of: Does online sustainability communication shape public discourse? Insights from six years of tenant-housing provider interactions},
year = {2026},
howpublished = {\url{https://pith.science/paper/NX6RCZBE}},
note = {Machine review of arXiv:2607.08437}
}
read the original abstract
Authorities increasingly rely on social media to advance sustainability transitions, infrastructure investment, and service reform. Yet how citizens respond to these digital communications remains poorly understood. Existing approaches rely on aggregate engagement metrics (e.g., likes), providing limited insight into discourse structure and quality. We developed a data-driven, multidimensional framework to analyse how social media communication shapes the content of discourse, focusing on sustainability-related engagement in Dutch public housing. We analysed 792 posts and 3,197 tenant comments from the Facebook pages of 92 housing providers (2018-2023). A machine-learning pipeline classified comments into recurring discourse configurations across three dimensions - communicative intent, sentiment, and semantic relatedness. Multinomial logistic regression estimated the effects of post-design and organisational characteristics on discourse. Tenant comments were significantly more semantically aligned with their corresponding posts than with randomly paired content, indicating that organisational communication structures responses to topics. Six discourse types emerged, with critical and inquiry-driven engagement increasing over time. Post-level features did not significantly explain variation; organisational characteristics dominated. Larger housing associations attracted more substantive responses, while lower-rent organisations received fewer evaluative comments. While applied to housing associations, our methodology provides a scalable approach to analyse online discourse dynamics, quality, and content across organisations and contexts.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Bartling, B., Valero, V ., Weber, R. A. & Yao, L. Public discourse and socially responsible market behavior.Am. Econ. Rev.114, 3041–3074, DOI: 10.1257/aer.20201361 (2024)
-
[2]
A.et al.Exposure to opposing views on social media can increase political polarization.Proc
Bail, C. A.et al.Exposure to opposing views on social media can increase political polarization.Proc. Natl. Acad. Sci.115, 9216–9221, DOI: 10.1073/pnas.1804840115 (2018). https://www.pnas.org/doi/pdf/10.1073/pnas. 1804840115
-
[3]
Zollo, S.et al.Examining the relationship between content length and engagement with news outlets on multiple social media platforms.Technol. Forecast. Soc. Chang.228, 124668, DOI: https://doi.org/10.1016/j.techfore.2026. 124668 (2026)
-
[4]
Nisar, T. M. & Prabhakar, G. Trains and twitter: Firm generated content, consumer relationship management and message framing.Transp. Res. Part A: Policy Pract.113, 318–334, DOI: https://doi.org/10.1016/j.tra.2018.04.026 (2018)
-
[5]
Nguyen, H., Drejer, I. & Marques, P. Citizen engagement in public sector innovation: exploring the transition between paradigms.Public Manag. Rev.26, 3622–3642, DOI: 10.1080/14719037.2024.2347360 (2024)
arXiv 2024
-
[6]
Liu, Z., Geng, R., Tse, Y . K. & Han, S. Mapping the relationship between social media usage and organizational performance: A meta-analysis.Technol. Forecast. Soc. Chang.187, 122253 (2023)
2023
-
[8]
Stone, J. A. & Can, S. H. Linguistic analysis of municipal twitter feeds: Factors influencing frequency and engagement.Gov. Inf. Q.37, 101468, DOI: https://doi.org/10.1016/j.giq.2020.101468 (2020)
arXiv 2020
-
[9]
Han, X. & Baird, C. Government crisis messaging on social media, citizen online engagement and compliance with policies.Public Manag. Rev.26, 613–634, DOI: 10.1080/14719037.2022.2103843 (2024)
arXiv 2022
Show all 43 references
-
[10]
& Evans, R
Zhang, W., Yuan, H., Zhu, C., Chen, Q. & Evans, R. Does citizen engagement with government social media accounts differ during the different stages of public health crises? an empirical examination of the covid-19 pandemic.Front. Public Heal.Volume 10 - 2022, DOI: 10.3389/fpub...
2022
-
[11]
Comments, shares, or likes: What makes news posts engaging in different ways.Soc
Tenenboim, O. Comments, shares, or likes: What makes news posts engaging in different ways.Soc. Media + Soc. 8, 20563051221130282, DOI: 10.1177/20563051221130282 (2022)
2022 doi
-
[12]
& Nair, H
Lee, D., Hosanagar, K. & Nair, H. S. Advertising content and consumer engagement on social media: Evidence from facebook.Manag. Sci.64, 5105–5131 (2018)
2018
-
[13]
Grimmelikhuijsen, S., Porumbescu, G., Hong, B. & Im, T. The effect of transparency on trust in government: A cross-national comparative experiment.Public Adm. Rev.73, 575–586, DOI: 10.1111/puar.12047 (2013)
2013 doi
-
[14]
& Gori, E
Contri, M., Fissi, S. & Gori, E. Using social media for supporting engagement in the public sector: a systematic literature review.Local Gov. Stud.51, 1–23, DOI: 10.1080/03003930.2023.2288942 (2025)
2023
-
[15]
& Wizner, S
Biber, D., Egbert, J., Keller, D. & Wizner, S. Towards a taxonomy of conversational discourse types: An empirical corpus-based analysis.J. Pragmat.171, 20–35, DOI: https://doi.org/10.1016/j.pragma.2020.09.018 (2021). 16.Romero-Trillo, J. Corpus-based approaches to discourse an...
2020 doi
-
[17]
Social media engagement forms in government: A structure-content framework.Gov
Wukich, C. Social media engagement forms in government: A structure-content framework.Gov. Inf. Q.39, 101684, DOI: https://doi.org/10.1016/j.giq.2022.101684 (2022)
2022
-
[18]
Laureate, C. D. P., Buntine, W. & Linger, H. A systematic review of the use of topic models for short text social media analysis.Artif. Intell. Rev.56, 1573–7462, DOI: 10.1007/s10462-023-10471-x (2023). 10/20
2023 doi
-
[19]
V ., Kerperien, S
Ossokina, I. V ., Kerperien, S. & Arentze, T. A. Does information encourage or discourage tenants to accept energy retrofitting of homes?Energy Econ.103, None, DOI: 10.1016/j.eneco.2021.105534 (2021). 20.V oorberg, W. H., Bekkers, V . J. & Tummers, L. G. A systematic review of...
2021
-
[21]
G., Pierre, J
Torfing, J., Peters, B. G., Pierre, J. & Sørensen, E.Interactive Governance: Advancing the Paradigm(Oxford University Press, 2012)
2012
-
[22]
Sharing feelings and user engagement on twitter: It’s all about me and you.Soc
Segev, E. Sharing feelings and user engagement on twitter: It’s all about me and you.Soc. Media + Soc.9, 20563051231183430, DOI: 10.1177/20563051231183430 (2023). https://doi.org/10.1177/20563051231183430
2023 doi
-
[23]
& Cantijoch, M
Gibson, R. & Cantijoch, M. Conceptualizing and measuring participation in the age of the internet: Is online political engagement really different to offline?The J. Polit.75, DOI: 10.1017/S0022381613000431 (2013)
2013 doi
-
[24]
& Burnay, C
Simonofski, A., Fink, J. & Burnay, C. Supporting policy-making with social media and e-participation platforms data: A policy analytics framework.Gov. Inf. Q.38, 101590, DOI: https://doi.org/10.1016/j.giq.2021.101590 (2021)
2021
-
[25]
& Ayres, L
Condie, J. & Ayres, L. Tenant participation and emerging social media practices in the social housing sector.Hous. Stud.40, 589–609, DOI: 10.1080/02673037.2024.2310701 (2025)
2024
-
[26]
& Gurevych, I
Reimers, N. & Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of EMNLP 2019(2019)
2019
-
[27]
R., Benitez, J
Li, K., Zhou, C., Luo, X. R., Benitez, J. & Liao, Q. Impact of information timeliness and richness on public engagement on social media during COVID-19 pandemic: An empirical investigation based on NLP and machine learning.Decis. Support. Syst.162, 113752, DOI: 10.1016/j.dss.2...
2022
-
[30]
R.Speech Acts: An Essay in the Philosophy of Language(Cambridge University Press, Cambridge, 1969)
Searle, J. R.Speech Acts: An Essay in the Philosophy of Language(Cambridge University Press, Cambridge, 1969)
1969
-
[31]
& Greenwood, B
Wang, S. & Greenwood, B. N. Too long, didn’t read? the impact of length limits on microblogging engagement. Manag. Sci.71, 9437–9461 (2025). 32.Hussin, Z. N., Chin, C. P.-Y . & Sondoh Jr, S. L. Engaging citizens on government social media page: The role of content characterist...
2025
-
[33]
& Lebherz, P
Sabate, F., Berbegal-Mirabent, J., Cañabate, A. & Lebherz, P. R. Factors influencing popularity of branded content in facebook fan pages.Eur. Manag. J.32, 1001–1011, DOI: https://doi.org/10.1016/j.emj.2014.05.001 (2014)
2014 doi
-
[34]
& Rutherford, A
Wallace, T. & Rutherford, A. C. The big bird gets the worm? how size influences social networking by charitable organizations.Nonprofit Volunt. Sect. Q.50, 626–646 (2021)
2021
-
[35]
& Saxton, G
Lovejoy, K. & Saxton, G. Information, community, and action: How nonprofit organizations use social media.J. Comput. Commun.17, 337–353, DOI: 10.1111/j.1083-6101.2012.01576.x (2012)
2012
-
[36]
Social media adoption and resulting tactics in the u.s
Mergel, I. Social media adoption and resulting tactics in the u.s. federal government.Gov. Inf. Q.30, 123–130 (2013)
2013
-
[37]
Ubani, S., Polat, S. O. & Nielsen, R. Zeroshotdataaug: Generating and augmenting training data with chatgpt (2023). 2304.14334. 38.de Vries, W.et al.BERTje: A Dutch BERT Model. arXiv:1912.09582 (2019)
2023 arXiv
-
[38]
Note: The comment should blame someone (criticize a particular subject/entity) for their actions or behavior
Criticism Definition:The author expresses dissatisfaction, disapproval, or negative judgment about a situation, person, organiza- tion, or decision. Note: The comment should blame someone (criticize a particular subject/entity) for their actions or behavior. Common Patterns: •...
-
[39]
& Berendt, B
Delobelle, P., Winters, T. & Berendt, B. RobBERT: a Dutch RoBERTa-based Language Model. InFindings of EMNLP 2020, DOI: 10.18653/v1/2020.findings-emnlp.292 (2020)
2020 doi
-
[40]
& Chen, W
He, P., Gao, J. & Chen, W. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.ArXivabs/2111.09543(2021). 11/20 9 Acknowledgements We acknowledge support from the Dutch Science Foundation (NWO) grant 403.19.230 and the Dut...
2021 arXiv
-
[41]
great,” “thank you,
Appreciation Definition:The author expresses approval, gratitude, or positive feedback towards a person, service, or decision. There should be an entity that is being praised. Common Patterns: • Positive tone and sentiment • Use of words like “great,” “thank you,” “happy,” “ni...
-
[42]
Common Patterns: • Informative or descriptive tone • Often factual or narrative Trigger Phrases & Words: • Beautiful flowers
Statement – Positive Definition:The author shares an observation, fact, or opinion with a positive tone, but does not praise any entity. Common Patterns: • Informative or descriptive tone • Often factual or narrative Trigger Phrases & Words: • Beautiful flowers. . . • This wor...
-
[43]
Trigger Phrases & Words: • This does not work
Statement – Negative Definition:The author shares an observation, fact, or opinion with a negative tone, but does not blame anyone. Trigger Phrases & Words: • This does not work. . . • This is not what I expected. • There’s a serious issue with. . . • Clearly something’s wrong
-
[44]
Trigger Phrases & Words: • It appears that
Statement – Neutral Definition:The author shares an observation, fact, or opinion in a neutral tone, without clear emotional bias. Trigger Phrases & Words: • It appears that. . . • This is what happened. . . • I noticed that. . . • As far as I can tell. . . 14/20
-
[45]
? • Can someone tell me
Inquiry Definition:Includes both: •Questions:Asking for information, clarification, or solution •Requests:Asking for action or help Common Patterns: • Question format (with or without a question mark) • Tone of curiosity or confusion • Implicit questions (indirect inquiries) T...
-
[46]
Criticism
Forward Definition:The author tags another person or shares the post to inform someone else. Sentiment is mostly neutral or positive. Trigger:Presence of a<PERSON>tag in the beginning of the comment implies Forward intent. 15/20 Appendix B. Synthetic Dataset generation A few-s...
-
[47]
Klopt toch niks van, na 3 jaar een woning? Een ander reageert zich suf en krijgt helemaal niks, na 8 jaar ingeschreven te staan en dus ook al 8 jaar reageert op lotingen
-
[48]
En daarna flink betalen aan de vve
Op een woning met achterstallig onderhoud. En daarna flink betalen aan de vve. Dat geintje hebben ze ook bij Lauwers gedaan. Resultaat. Alles mooi opgeknapt behalve de verkochte flat
-
[49]
Comments were generated iteratively
Mijn huis zit vol schimmel me kleding schoenen alles in huis gaat er aan we zijn ook altijd ziek Similarly, comments were generated for other intents, exceptForward, by replacing the intent in the prompt, along with examples for reference (no posts were inserted in the prompt)...
2023
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.