Pith. sign in

REVIEW 3 major objections 4 minor 20 references

Building a Media Ecosystem Observatory from Scratch: Infrastructure, Methodology, and Insights

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper presents the Canadian Media Ecosystem Observatory, a functioning national-scale infrastructure that collects, normalizes, and semantically indexes political and media discourse across X, Instagram, TikTok, Facebook, Bluesky…

desk verdict A solid, honest infrastructure paper whose headline 'near real-time' claim is contradicted by its own crawler schedule; the empirical Meta ban numbers need methodology before they should be cited. read the letter →

arxiv 2506.10942 v1 pith:SIL7CHHM submitted 2025-06-12 cs.DL

Zeynep Pehlivan , Saewon Park , Alexei Sisulu Abrahams , Mika Desblancs-Patel , Benjamin David Steel , Aengus Bridgman This is my paper
classification cs.DL
keywords mediaobservatorycross-platformanalysissocialmonitoringinformationecosystemdigitaltracedatasemanticembeddingsseedlistMetanewsban
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a national-scale, cross-platform observatory can be built from scratch even as platform APIs become restricted, and that such a system can produce timely, policy-relevant findings. It presents the Canadian Media Ecosystem Observatory, which combines platform-specific crawlers, a normalization layer that maps heterogeneous schemas into a common data model, semantic embeddings, and API/dashboard access. The authors show the system already works by measuring a roughly 64% drop in engagement on national news outlets' Facebook Pages and 85% for local outlets after Meta's 2023 news ban, and by powering weekly situation reports during the 2025 federal election. If the infrastructure performs as described, it offers a replicable model for digital trace research in other national contexts.

What carries the argument

The central mechanism is the seed-list-anchored crawl-and-normalize pipeline: a curated list of 5,515 entities defines what is collected, custom crawlers for each platform retrieve posts, and a normalization layer harmonizes schemas into a single common model while preserving raw JSON for future re-indexing. Semantic embeddings stored in a vector database add similarity search and topic clustering, and a monitoring layer tracks crawler health and coverage gaps. The seed list itself is the load-bearing object, since it determines which actors and posts enter the dataset.

What would settle it

Recompute the Meta-news-ban engagement drop after backfilling Facebook data from August 2024 to January 2025 and after adding municipal politicians from smaller cities; if the roughly 64% national and 85% local engagement drops move materially, the reported effect was an artifact of coverage rather than the ban itself.

Watch

Extended reading notes

Core claim

The central claim is that the Canadian Media Ecosystem Observatory is a working, modular, and scalable infrastructure for monitoring political and media discourse across multiple platforms in near real time. Its 5,515-entity seed list anchors collection of posts from politicians, news outlets, influencers, government bodies, civil society organizations, and select foreign actors; platform-specific crawlers feed raw JSON to cloud storage; a normalization layer maps each platform's fields into a common schema; transformer-based embeddings enable semantic search and clustering; and RESTful APIs plus dashboards provide programmatic and visual access. The paper's demonstrations are that the system captured the effects of Meta's news ban on Canadian news engagement and supported topic tracking during the 2025 federal election.

Load-bearing premise

The system's findings inherit whatever bias exists in its seed list of 5,515 entities, which prioritizes the most visible national and provincial actors, excludes municipal politicians from cities under 100,000, applies follower thresholds to define influencers, and loses Facebook coverage after August 2024.

Editorial extensions

If this is right

  • If the infrastructure works as claimed, researchers can conduct cross-platform analyses of Canadian political discourse without building their own collection stack, and the model can be adapted to other countries by generating a new seed list and customizing metadata schemas.
  • The Meta news ban analysis shows that the system can quantify platform policy effects: national news outlets' Facebook engagement fell roughly 64%, local outlets' by 85%, and nearly half of local outlets stopped posting within four months of the ban.
  • The semantic indexing layer enables topic modeling, semantic clustering, and similarity search across platforms over time, as demonstrated by the federal election situation reports.
  • Storing raw JSON alongside normalized data means the dataset can be re-indexed and re-analyzed as research questions and schemas evolve, supporting reproducibility and longitudinal study.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The entity-centric design means that events driven by actors outside the seed list, or by hashtags and keywords rather than accounts, are not captured; the authors note they are adding event-based crawlers, which would complement the current approach.
  • Because Facebook collection paused in August 2024, the reported Meta-ban statistics cover only the period before the pause; if the planned backfill succeeds, the medium-term effects of the ban could be reassessed and the engagement drops might change.
  • A testable extension would be to compare observatory engagement metrics with independent platform-wide measurements from restricted-access tools or third-party panels, which would quantify any bias introduced by the seed-list coverage decisions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes the Canadian Media Ecosystem Observatory, a multi-platform data collection and analysis infrastructure. It details the construction of a seed list of 5,515 Canadian political, media, influencer, government, civil-society, and foreign actors; custom crawlers for X/Twitter, TikTok, Instagram, Facebook, Bluesky, and YouTube; an indexing and normalization pipeline that integrates raw JSON into per-platform and unified Elasticsearch indices; a semantic embedding and vector-search layer; and APIs, dashboards, and a monitoring system. The authors present exploratory statistics on post volumes and two illustrative applications: the August 2023 Meta news ban and the 2025 Canadian federal election. The central claim is that the observatory provides near-real-time, cross-platform data that supports both automated and ad hoc research.

Significance. If the infrastructure performs as described, it would be a valuable and replicable resource for Canadian political communication research, offering an unusually broad multi-platform coverage that includes TikTok, Bluesky, and Telegram in addition to the usual X/Facebook pair. The paper's strengths are the detailed architectural description, the explicit discussion of platform-access challenges such as the CrowdTangle shutdown, and the inclusion of a crawled-date field that records collection timing independently of platform timestamps. The empirical claims, however, are illustrative and are not backed by a methodology section, and the 'near real-time' framing is contradicted by the reported collection frequencies. With revision, the paper could serve as a useful infrastructure report.

major comments (3)
  1. [Section 3.2 (Media Crawlers) vs. Abstract and Section 3] The claim that the observatory operates 'in near real time' (Abstract) and supports 'almost real-time (d-3)' processing (Section 3) is not supported by the crawler schedules described in Section 3.2: X/Twitter is collected on a 'bi-weekly' basis, TikTok 'at regular weekly intervals,' Instagram post-CrowdTangle 'on a weekly basis,' and Facebook collection was paused after August 14, 2024. With weekly collection, a post can be up to seven days old before it enters the index, so a stated three-day (d-3) latency cannot hold for TikTok or Instagram, which contribute a substantial share of the data in Table 2. Because Section 2 explicitly distinguishes this observatory from the German SMO by its 'greater emphasis on near real-time collection,' this inconsistency is load-bearing for the paper's claimed novelty. Please define what 'd-3' means, report actual collection-to-index latency distributions per platform, and either increase collection frequencies or revise the near-real-time claims to 'periodic batch collection.'
  2. [Section 6 (Applications, Meta News Ban)] The quantitative findings in this section — a '64%' engagement drop for national outlets, an '85%' drop for local outlets, 'nearly half' of local outlets stopping posting within four months, and a decline in misinformation links — are presented without any methodology. There is no definition of the engagement metric, no stated baseline or comparison window, no sample size or unit of analysis, no confidence intervals, and no discussion of how the August 2024 pause in Facebook collection affects the post-ban analysis window. Since these numbers are the primary 'insights' demonstrating the observatory's utility, the paper needs to include a methods paragraph (or a detailed pointer to reference [10] explaining exactly how these figures were computed) so readers can assess their validity and scope.
  3. [Section 3.2 (Media Crawlers) and Tables 1-2] The crawler section describes collection methods for X/Twitter, TikTok, Instagram, Facebook, Bluesky, and YouTube, but Tables 1 and 2 also report substantial Telegram data: 89 seed entities and over 1.9 million foreign posts. No Telegram collection mechanism is described anywhere in the paper. Because the paper's contribution is a complete infrastructure description, this omission leaves a significant gap. Either add a subsection describing the Telegram scraper and its seed-list sourcing, or explicitly state why Telegram is included in the tables and excluded from the crawler description.
minor comments (4)
  1. [Section 3.2] The term 'bi-weekly' is ambiguous; please state explicitly whether it means twice a week or every two weeks, since this directly affects the near-real-time claim.
  2. [Table 1] The table title is 'Distribution of Entities Across Platforms,' but the percentages appear to be computed within actor-type columns (e.g., 81.0% of news entities are on X/Twitter). Please state the denominator in the caption so readers can interpret the percentages correctly.
  3. [Figure 3] The text says the timeline represents 'data collected over two years,' but the collection period is described elsewhere as running from early 2022 to the 2025 election; please clarify the exact date range shown in the figure.
  4. [Section 4.1] The sentence 'visual dashboards were created using Kibana12 2' contains a stray superscript '2'; please remove it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a self-contained infrastructure description whose illustrative analyses are observational uses of its own collected data, not derivations from fitted inputs.

full rationale

This paper is a systems and methodology report on building the Canadian Media Ecosystem Observatory. It makes no theoretical derivation, fits no parameters, and presents no prediction that is forced by constructing its inputs. The illustrative analyses (Meta news ban engagement drops, election topic tracking) are descriptive uses of the observatory's own dataset, which is an observational workflow rather than a circular argument. The only self-citations are [1], [10], and [15]: [1] and [15] describe scraper approaches adopted for Instagram and TikTok collection, and [10] is the team's earlier preliminary report on the Meta ban. None of these is a load-bearing premise of the current paper's central claim, which is that the infrastructure exists, collects cross-platform data, normalizes it, and provides API/dashboard access. Even if [10] were entirely dependent on the same observatory data, the current paper does not rest its architecture claim on that report; the data itself is the object being described, not an unverified external theorem. The paper's own limitation section (7.1) acknowledges coverage windows differ across platforms, scrapers are less stable, and Facebook collection paused after August 2024, and the crawler cadences in Section 3.2 (bi-weekly X/Twitter, weekly TikTok, weekly Instagram post-CrowdTangle) conflict with the abstract's 'near real time' and Section 3's 'almost real-time (d-3)' framing. That is an internal consistency or correctness concern about claimed latency, not a circularity: no result is equivalent to its input by construction, and no uniqueness or ansatz is smuggled in via self-citation. Per the review rules, the limitation passages are weighed, but they do not create a derivation loop. The circularity burden is minimal and the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's contribution is infrastructure building, so the ledger is dominated by methodological assumptions about data coverage and comparability rather than mathematical postulates. There are no fitted theoretical constants.

free parameters (3)
  • Influencer follower thresholds = 10,000 on TikTok/X; 5,000 on Facebook/Instagram/YouTube/Bluesky
    Hand-selected inclusion criteria with no published justification; they directly determine which accounts enter the dataset and thus shape all downstream analyses.
  • Following-to-followers ratio threshold = 1:2
    Hand-chosen threshold for influencer identification; affects the influencer subset of the seed list.
  • CSO scope thresholds = not specified in text
    The paper mentions revenue and membership thresholds for civil society organizations without giving the numbers, making the seed list criteria partially underspecified.
assumptions (4)
  • domain assumption The 5,515-entity seed list provides representative coverage of Canadian political and media discourse.
    Invoked in Section 3.1; the paper explicitly excludes municipal politicians from cities under 100,000 and prioritizes high-visibility actors, so representation of the full ecosystem is assumed, not demonstrated.
  • domain assumption Covered platforms account for essentially all significant Canadian political conversations.
    Stated in Section 3.2: 'To a first approximation, all the major conversations and communities in Canadian political discourse are represented...' No external data is cited to support this.
  • domain assumption Custom scrapers return complete and unbiased post metadata.
    Section 7.1 admits scraping may miss posts due to rate caps and search index limits; the assumption that missing data is not systematically biasing is necessary for the validity of downstream engagement analyses.
  • domain assumption Engagement metrics are comparable across platforms after normalization.
    Section 4 describes normalization of heterogeneous schemas to a common model, but no validation is reported that normalized engagement figures (e.g., likes on TikTok vs. Facebook) measure comparable constructs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Building a Media Ecosystem Observatory from Scratch: Infrastructure, Methodology, and Insights." pith.science (2026). https://pith.science/paper/SIL7CHHM

@misc{pith2026250610942,
  author       = {Pith},
  title        = {Pith review of: Building a Media Ecosystem Observatory from Scratch: Infrastructure, Methodology, and Insights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SIL7CHHM}},
  note         = {Machine review of arXiv:2506.10942}
}
read the original abstract

Understanding the flow of information across today's fragmented digital media landscape requires scalable, cross-platform infrastructure. In this paper, we present the Canadian Media Ecosystem Observatory, a national-scale infrastructure designed to monitor political and media discourse across platforms in near real time. Media Ecosystem Observatory (MEO) data infrastructure features custom crawlers for major platforms, a unified indexing pipeline, and a normalization layer that harmonizes heterogeneous schemas into a common data model. Semantic embeddings are computed for each post to enable similarity search and vector-based analyses such as topic modeling and clustering. Processed and raw data are made accessible through API, dashboards and website, supporting both automated and ad hoc research workflows. We illustrate the utility of the observatory through example analyses of major Canadian political events, including Meta's 2023 news ban and the recent federal elections. As a whole, the system offers a model for digital trace infrastructure and an evolving research platform for studying the dynamics of modern media ecosystems.

Figures

Figures reproduced from arXiv: 2506.10942 by the authors.

Figure 1
Figure 1. Media Ecosystem Observatory Overall Architecture – Media Crawlers: gathers data from different platforms – Data Indexing: handles raw data ingestion, transformation, and backup; – Data Access: supports automated and ad-hoc analysis pipelines and data access, provides access to the processed information through dashboards and export tools; – Monitoring Layer: tracks system performance and data and pipeline health. Ea… view at source ↗
Figure 2
Figure 2. Kibana dashboard example 4.2 Monitoring Layer To ensure the reliability and stability of our data collection and processing pipeline, the observatory includes a dedicated monitoring layer. Each component in the system—including media crawlers, data normalization scripts, and index￾ing jobs—logs its status and performance metrics to Amazon CloudWatch14 . This centralized logging infrastructure allows for real-time tr… view at source ↗
Figure 3
Figure 3. Timeline by platform we present selected examples that illustrate how the system has been used to monitor political communication, media coverage, and cross-platform dynamics. Meta News Ban In response to the Canadian government’s Online News Act, Meta began blocking news content from its platforms for Canadian users on August 9, 2023. This policy change, referred to here as “Meta ban”, removed the visibility of all… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Topic analysis during federal elections 2025 More recently, we’ve leveraged semantic embeddings and vector search to sup￾port exploratory research. Each post is embedded as a dense vector, enabling researchers to retrieve semantically similar content, detect narrative …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 15 canonical work pages

  1. [10]

    https://doi.org/10.31235/osf.io/eqn45, https://osf.io/eqn45

    Parker, S., Park, S., Pehlivan, Z., Abrahams, A., Desblancs, M., Owen, T., Phillips, J., Bridgman, A.: When journalism is turned off: Prelim- inary findings on the effects of meta’s news ban in canada (2024). https://doi.org/10.31235/osf.io/eqn45, https://osf.io/eqn45

    work page doi:10.31235/osf.io/eqn45 2024

  2. [1]

    Abrahams, A.S.: Social media observatory (2023), https://nostarch.com/social-media-observatory

    work page 2023

  3. [2]

    Proceedings of the International AAAI Conference on Web and Social Media18(1), 1872– 1887 (May 2024)

    Ai, L., Gupta, S., Oak, S., Hui, Z., Liu, Z., Hirschberg, J.: Tweetintent@crisis: A dataset revealing narratives of both sides in the russia-ukraine crisis. Proceedings of the International AAAI Conference on Web and Social Media18(1), 1872– 1887 (May 2024). https://doi.org/10.1609/icwsm.v18i1.31432, https://ojs.aaai.org/index.php/ICWSM/article/view/31432

    work page doi:10.1609/icwsm.v18i1.31432 2024

  4. [3]

    https://doi.org/10.1080/10584609.2022.2128949, https://doi.org/10.1080/10584609.2022.2128949, publisher: Routledge _eprint: https://doi.org/10.1080/10584609.2022.2128949

    Bossetta, M., Schmøkel, R.: Cross-platform emotions and audience engagement in social media political campaigning: Comparing can- didates’ facebook and instagram images in the 2020 US election 40(1), 48–68 (2023). https://doi.org/10.1080/10584609.2022.2128949, https://doi.org/10.1080/10584609.2022.2128949, publisher: Routledge _eprint: https://doi.org/10....

    arXiv 2023

  5. [4]

    Brown, M.A., Gruen, A., Maldoff, G., Messing, S., Sanderson, Z., Zimmer, M.: Web scraping for research: Legal, ethical, institutional, and scientific considerations (2024), https://arxiv.org/abs/2410.23432

    arXiv 2024

  6. [5]

    In: Han, H., Baker, E

    Dogdu, E., Choupani, R., Sürücü, S.: Detecting Political Polarization Using Social Media Data. In: Han, H., Baker, E. (eds.) Next Genera- tion Data Science. pp. 46–59. Springer Nature Switzerland, Cham (2024). https://doi.org/10.1007/978-3-031-61816-1_4

    work page doi:10.1007/978-3-031-61816-1_4 2024

  7. [6]

    arXiv preprint arXiv:2203.05794 (2022)

    Grootendorst, M.: Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794 (2022)

    arXiv 2022

  8. [7]

    Digital Journalism11(8), 1499– 1520 (Sep 2023)

    Hase, V., , Karin, B., , Scharkow, M.: Adapting to Affor- dances and Audiences? A Cross-Platform, Multi-Modal Anal- ysis of the Platformization of News on Facebook, Insta- gram, TikTok, and Twitter. Digital Journalism11(8), 1499– 1520 (Sep 2023). https://doi.org/10.1080/21670811.2022.2128389, https://doi.org/10.1080/21670811.2022.2128389

    arXiv 2023

Show all 20 references
  1. [8]

    In- ternational Journal of Communication15(0), 22 (Jan 2021), https://ijoc.org/index.php/ijoc/article/view/15042

    Matassi, M., Boczkowski, P.: An Agenda for Comparative So- cial Media Studies: The Value of Understanding Practices From Cross-National, Cross-Media, and Cross-Platform Perspectives. In- ternational Journal of Communication15(0), 22 (Jan 2021), https://ijoc.org/index.php/ijoc/...

    2021

  2. [9]

    JMIR Public Health and Surveillance7(11), e30642 (Nov 2021)

    Muric, G., Wu, Y., Ferrara, E.: COVID-19 Vaccine Hesitancy on So- cial Media: Building a Public Twitter Data Set of Antivaccine Con- tent, Vaccine Misinformation, and Conspiracies. JMIR Public Health and Surveillance7(11), e30642 (Nov 2021). https://doi.org/10.2196/30642, http...

    2021 doi

  3. [11]

    In: The past web: Exploring web archives, pp

    Pehlivan, Z., Thièvre, J., Drugeon, T.: Archiving social media: the case of twitter. In: The past web: Exploring web archives, pp. 43–56. Springer (2021)

    2021

  4. [12]

    Designs3(2), 24 (Jun 2019)

    Perakakis, E., Mastorakis, G., Kopanakis, I.: Social Media Monitor- ing: An Innovative Intelligent Approach. Designs3(2), 24 (Jun 2019). https://doi.org/10.3390/designs3020024, https://www.mdpi.com/2411- 9660/3/2/24

    2019 doi

  5. [13]

    New Media & Society25(11), 3165–3188 (Nov 2023)

    Ronzhyn, A., Cardenal, A.S., Batlle Rubio, A.: Defining affordances in social media research: A literature review. New Media & Society25(11), 3165–3188 (Nov 2023). https://doi.org/10.1177/14614448221135187, https://doi.org/10.1177/14614448221135187

    2023 doi

  6. [14]

    In: Proceedings of the 25th International Conference Companion on World Wide Web - WWW ’16 Companion

    Shao, C., Ciampaglia, G.L., Flammini, A., Menczer, F.: Hoaxy: A platform for tracking online misinformation. In: Proceedings of the 25th International Conference Companion on World Wide Web - WWW ’16 Companion. p. 745–750. WWW ’16 Compan- ion, ACM Press (2016). https://doi.org...

    2016

  7. [15]

    Steel, B., Parker, S., Ruths, D.: The invasion of ukraine viewed through large-scale analysis of tiktok (2024)

    2024

  8. [16]

    https://doi.org/10.1016/j.techfore.2023.123197, https://www.sciencedirect.com/science/article/pii/S004016252300882X

    Sun, Y., Jia, R., Razzaq, A., Bao, Q.: Social network plat- forms and climate change in china: Evidence from TikTok200, 123197 (2024). https://doi.org/10.1016/j.techfore.2023.123197, https://www.sciencedirect.com/science/article/pii/S004016252300882X

    2024

  9. [17]

    Publizistik68(2), 201– 223 (Sep 2023)

    Wiedemann, G., Münch, F.V., Rau, J.P., Kessling, P., Schmidt, J.H.: Concept and challenges of a social media observa- tory as a DIY research infrastructure. Publizistik68(2), 201– 223 (Sep 2023). https://doi.org/10.1007/s11616-023-00807-6, https://doi.org/10.1007/s11616-023-00807-6

    2023 doi

  10. [18]

    Journal of Computational Social Sci- ence5, 1511–1528 (2022)

    Yang, K.C., Ferrara, E., Menczer, F.: Botometer 101: Social bot practicum for computational social scientists. Journal of Computational Social Sci- ence5, 1511–1528 (2022). https://doi.org/10.1007/s42001-022-00177-5, https://rdcu.be/cUvyT

    2022 doi

  11. [19]

    https://doi.org/10.48550/arXiv.2407.02807, http://arxiv.org/abs/2407.02807, arXiv:2407.02807 [cs]

    Yang, Z., Imouza, A., Touzel, M.P., Amadoro, C., Desrosiers- Brisebois, G., Pelrine, K., Levy, S., Godbout, J.F., Rabbany, R.: Regional and Temporal Patterns of Partisan Polarization during the COVID-19 Pandemic in the United States and Canada (Jul 2024). https://doi.org/10.48...

    2024 doi arXiv

  12. [20]

    teractional, positional, and affective polarization on social media 38(1), 98–139 (2021)

    Yarchi, M., Baden, C., Kligler-Vilenchik, N.: Political polarization on the digital sphere: A cross-platform, over-time analysis of in- 18 Z.Pehlivan et al. teractional, positional, and affective polarization on social media 38(1), 98–139 (2021). https://doi.org/10.1080/105846...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.