Improving Regulatory Oversight in Online Content Moderation

Anna Monreale; Benedetta Tessa; Denise Amram; Stefano Cresci

REVIEW 3 major objections 5 minor 20 references

Improving Regulatory Oversight in Online Content Moderation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes two complementary processes—a Transparency Report Cross-Checking Process and a Verification Process—to detect inconsistencies between platforms' self-reported and actual moderation data under the EU Digital Services Act.

desk verdict A useful design sketch for DSA transparency auditing; the cross-check process is sound, but the verification process relies on AI labels as ground truth, which is an unproven and likely problematic assumption. read the letter →

arxiv 2506.04145 v1 pith:V5TEWV2D submitted 2025-06-04 cs.CY

Benedetta Tessa , Denise Amram , Anna Monreale , Stefano Cresci This is my paper

classification cs.CY

keywords DigitalServicesActcontentmoderationtransparencyreportsDSADatabaseplatformgovernanceaccountabilityAIclassificationregulatoryoversight

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes two complementary processes to verify the reliability of transparency data that large platforms must report under the EU Digital Services Act (DSA): a Transparency Report Cross-Checking Process and a Verification Process. The first compares the aggregate statistics in platforms' Transparency Reports against aggregations computed from the DSA Transparency Database (DSA-TDB) records, checking the internal coherence of self-reported data. The second compares DSA-TDB Statements of Reasons with actual platform content and metadata obtained under Article 40, using AI to classify moderated content and reconstruct the statements for external validation. The paper argues that together these processes would let regulators, researchers, and platforms themselves detect inconsistencies such as platforms reporting automated moderation in one place but none in another. A sympathetic reader would care because the DSA's transparency tools currently fail to answer basic questions about content moderation, and these processes are a concrete route to making self-reported data checkable.

What carries the argument

The central objects are the two named processes. The Transparency Report Cross-Checking Process extracts aggregations from unstructured Transparency Reports (PDFs, web pages, charts) via OCR, rule-based parsing, or large language models, replicates those aggregations from DSA-TDB Statements of Reasons filtered by attributes such as violation type and date, and compares the two sets of numbers. The Verification Process extracts moderated content and metadata from platform data obtained under Article 40, uses AI classifiers (natural language processing and image recognition) to label content by violation type, reconstructs Statements of Reasons, and compares them with the DSA-TDB records. Their complementarity is the mechanism: one checks coherence between two self-reported sources, the other checks the truthfulness of the self-reported record against actual platform data.

What would settle it

Run the Verification Process on a set of platform moderation records where the true violation labels are known (for example, data obtained through an Article 40 request that includes the platform's own annotations). If the AI-reconstructed Statements of Reasons differ from the platform's internal labels about as often as they differ from the DSA-TDB records, the process cannot separate classifier error from platform misreporting, and the comparison baseline collapses.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that the DSA's transparency components—the DSA-TDB, Transparency Reports, and Article 40 data access—are individually valuable but unverifiable, and that layering two dedicated validation processes on top of them can turn self-reported moderation data into auditable evidence. The Transparency Report Cross-Checking Process treats the DSA-TDB as the granular record and Transparency Reports as the aggregate summary; if the two cannot be reconciled, the discrepancy flags the report for investigation, without needing to know which source is wrong. The Verification Process treats platform data accessed through Article 40 as ground truth: AI tools classify moderated content by violation type, metadata is used to reconstruct Statements of Reasons according to DSA guidelines, and those reconstructions are compared with the original DSA-TDB records to expose omissions, misclassifications, and selective reporting. The paper does not report a working implementation or accuracy figures; its contribution is the design of the two processes and the argument that their combination creates internal and external validation loops the DSA currently lacks.

Load-bearing premise

The Verification Process assumes that AI algorithms can label moderated content by violation type accurately enough that the reconstructed Statements of Reasons are a trustworthy baseline for comparison; the paper gives no accuracy estimates or error analysis, and if classifiers disagree with platforms' internal labels, detected discrepancies become ambiguous.

Editorial extensions

If this is right

Regulators could automatically flag Transparency Reports whose headline statistics cannot be reproduced from DSA-TDB records, turning formal compliance checking into a data-driven audit.
External validation would expose gaps like platforms reporting automated moderation in Transparency Reports while filing no automated actions in the DSA-TDB, giving the European Commission concrete evidence for proceedings.
Platforms could run the Cross-Checking Process internally before publication, reducing inadvertent inconsistencies and raising the quality of self-reported data.
Reconstructed Statements of Reasons could support shared certification standards for what counts as trustworthy reporting under the DSA, and inform delegated acts on Article 40 data access.
Researchers using the DSA-TDB would gain a way to assess whether the database is complete enough to answer the kinds of questions it was designed for.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves untested whether LLM-based extraction is accurate enough on real Transparency Reports; a natural pilot would benchmark extraction against a sample of manually read reports, since extraction errors and platform misreporting are observationally confounded.
The AI classification step's reliance on violation-type labels suggests a testable dependency: classifiers trained on open datasets may not match platforms' internal enforcement categories, so useful verification requires aligning taxonomies first.
A direct extension would be to run the Cross-Checking Process on already-published reports and DSA-TDB snapshots (for example, for X, Facebook, and Instagram) to quantify how many aggregate statistics currently fail to reconcile.
The Verification Process could be generalized beyond content moderation to other DSA obligations, such as recommender-system transparency and advertising archives, wherever a self-reported aggregate can be compared with underlying logs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Reading this paper, the useful thing is the separation of two validation tasks. The Transparency Report Cross-Checking Process is honestly framed: both sources are self-reported, so it checks internal coherence, not truth. That is a sensible, implementable idea, and the OCR/LLM extraction pipeline is a reasonable assembly of existing tools. The Verification Process, however, has a load-bearing gap. It proposes reconstructing Statements of Reasons by having AI classifiers label moderated content, then comparing those to the platform's SoRs. But the AI labels are not an independent ground truth. Hate speech, misinformation, and nudity are contested, context-dependent, and platform-specific categories. A mismatch can be classifier error, a taxonomy difference, or a legitimate judgment difference, not misreporting. The paper gives no accuracy estimates, no error analysis, and no plan to handle the generic 'other' category. So the central efficacy claim in the abstract—detecting inconsistencies between self-reported and actual platform data—is not supported by the described design. The paper is a position/design piece and does not claim empirical results, which limits the damage. But as a design, the verification step needs rework: either frame it as generating leads for manual auditing rather than automated verification, or build in calibration against platform metadata and human review, or position AI labels as a source of discrepancies to investigate rather than as ground truth.

The legal implications section is mostly generic and could be tightened, but the core insight that consistency checks matter for enforcement and codes of conduct is sound. The authors cite their own prior audits of the DSA-TDB, which is appropriate here because those audits motivate the problem and are documented. The citation pattern is fine.

Who is this for? Regulators, DSA researchers, and platform governance people thinking about audit tooling. It is a legitimate design sketch, useful as a starting point for a pilot, not as evidence that such verification works. I would send it to peer review with the expectation of major revision: soften the central claim, address the ground-truth problem in the Verification Process, and ideally add a small case study on one platform and one period to show the cross-check process can actually surface discrepancies.

Referee Report

3 major / 5 minor

Summary. The paper argues that the EU Digital Services Act's transparency instruments—the DSA Transparency Database (DSA-TDB), Transparency Reports, and Article 40 data access—currently fail to deliver reliable oversight because platforms self-report inconsistently and with low detail. It proposes two complementary processes: a Transparency Report Cross-Checking Process, which compares aggregations extracted from Transparency Reports with aggregations computed from DSA-TDB records, and a Verification Process, which compares DSA-TDB Statements of Reasons with platform data obtained under Article 40, using AI classifiers to reconstruct the Statements of Reasons. The authors claim these processes provide internal and external validation, detect inconsistencies between self-reported and actual platform data, and thereby improve transparency, compliance assessment, and accountability under the DSA. No implementation, pilot study, or empirical evaluation is reported.

Significance. If the proposed processes worked as claimed, they would give regulators and researchers a systematic way to audit the coherence of DSA-mandated reports and to surface mismatches between platforms' submissions and their actual moderation records. The paper's strengths are its clear problem motivation, its careful cataloging of documented data-quality problems in the DSA-TDB and Transparency Reports, and its useful distinction between internal consistency checks and external validation. It also makes the falsifiable procedural claim that such checks can be built from existing data sources. However, the central efficacy claim is not yet supported: the Verification Process depends on untested assumptions about AI classification quality and about the independence of platform-provided data, and the paper contains no proof-of-concept or error analysis. The contribution is currently a design sketch rather than a validated method.

major comments (3)

[Section 3.1.2 (Verification Process)] The central efficacy claim that the Verification Process can 'detect possible inconsistencies between self-reported and actual platform data' depends on treating AI labels as a valid proxy for the platform's true moderation reasons. The paper provides no accuracy estimates, baselines, or error analysis for the proposed multimodal classifiers, and the contested, platform-specific nature of categories such as hate speech, misinformation, and nudity means that a mismatch between a reconstructed and an original Statement of Reasons is ambiguous: it could reflect classifier error, a different taxonomy, or a legitimate difference in judgment rather than non-compliance. The paper also does not explain how the frequent generic 'other' category in the DSA-TDB would be reconciled with AI labels. Please add concrete validation evidence, discuss error and uncertainty propagation, or explicitly narrow the claimed capability.
[Section 3.1.2 and Figure 1] The process is described as external validation because it compares Statements of Reasons with 'actual platform data' obtained under Article 40. However, Article 40 access is granted by the platform itself and can be partial, delayed, or curated, as the paper itself documents in Section 2 when discussing researchers' access difficulties. A comparison against platform-provided data is therefore still a form of self-reported validation and cannot by itself establish truthfulness or completeness. Please address how the process would distinguish a genuine inconsistency from a data-access artifact, or reframe the contribution as a semi-external consistency check rather than a validation against ground truth.
[Section 3.1.1 (Transparency Report Cross-Checking Process)] The Cross-Checking Process assumes that aggregations in Transparency Reports and aggregations computed from DSA-TDB records are directly comparable. This requires a shared definition of time window, content type, violation category, and counting unit (e.g., moderation actions vs. pieces of content), but the paper itself documents the absence of standardized reporting structures across platforms in Section 2. Without explicit reconciliation rules for these axes, any flagged discrepancy is uninterpretable as an inconsistency. Please specify how matching and normalization would be performed, or restrict the claim to cases where platforms use identical definitions.

minor comments (5)

[Section 3.1.2 and Figure 1] The phrase 'actual platform data' is misleading when the data are supplied by the platform under Article 40; consider using 'platform-provided data' consistently, including in the label in Figure 1.
[Section 3.1.2] Reconstructing a full Statement of Reasons requires fields such as the free-text statement and the legal basis, but the paper does not describe how these would be recovered from content metadata alone, so the reconstructed Statements of Reasons can only be partial records.
[Section 3.2] The sentence beginning 'These premises strongly confirm whether data included in the Transparency Reports and in theDSA-TDB are accurate and homogeneous' is grammatically incomplete and its claim is stronger than the preceding discussion supports; rephrase it as something like 'These premises highlight the importance of ensuring that the data... are accurate and homogeneous.'
[Author affiliation footnote] The author footnote contains garbled characters in the email address field ('envel⌢pe-⌢pen'); please fix this formatting artifact.
[Figure 1] The diagram is helpful, but the arrow from 'Platform shared data' to the completeness and trustworthiness check should be annotated with the Article 40 access condition, since the diagram otherwise implies unmediated access to platform records.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a design proposal whose two verification processes compare independent data sources; self-citations are motivational, not load-bearing.

full rationale

This paper proposes two complementary oversight processes rather than deriving a quantitative result, so most circularity patterns do not apply. The Transparency Report Cross-Checking Process compares aggregate statistics extracted from Transparency Reports with aggregations computed from DSA-TDB Statements of Reasons; these are two distinct, independently generated data sources, and any discrepancy is a genuine cross-check rather than an identity. The Verification Process compares platform-shared content and metadata with the DSA-TDB SoRs; even though the platform provides both data streams, the comparison is not a tautology because the DSA-TDB record and the underlying content metadata are different artifacts, and the reconstructed SoR is assembled from metadata and AI classification rather than copied from the original SoR. No equation defines an output in terms of the target quantity, no fitted parameter is later relabeled as a prediction, and no uniqueness theorem is imported from prior work. The authors do cite their own prior audits of the DSA-TDB (refs [2] and [8]) to motivate the existence of data inconsistencies, but those citations are independently checkable analyses of a public database and are not used to establish that the proposed processes work. The skeptical concern about AI classification accuracy and the absence of a ground truth for violation labels is a substantive feasibility and validity limitation, but it is not circularity: an AI-reconstructed SoR could disagree with a platform SoR because of classifier error, taxonomy differences, or genuine misreporting, but the comparison itself is not equivalent to its inputs by construction. The paper makes no derivation chain that reduces to its own premises, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities are introduced. The central proposal rests on four domain assumptions about data comparability, data access, AI classification accuracy, and category alignment, none of which are empirically validated in the paper.

assumptions (4)

domain assumption DSA-TDB records can be aggregated to reproduce the statistics in Transparency Reports.
Section 3.1.1 assumes SoRs include attributes such as violation type, date, and content type that allow filtering and matching each transparency report aggregate. The paper itself documents that platforms often use generic categories and omit optional fields, so this alignment may fail in practice.
domain assumption Article 40 grants access to platform data containing content metadata sufficient to reconstruct Statements of Reasons.
Section 3.1.2 relies on retrieving visibility status, violation categories, creation dates, and platform-specific flags from platform data. Section 2 notes that Article 40 access is slow, vague, and restrictive, so this premise is uncertain.
domain assumption AI content classifiers can accurately classify moderated content across violation types.
Section 3.1.2 proposes AI algorithms for hate speech, misinformation, deepfakes, and nudity classification without providing accuracy estimates or baselines. Since reconstructed SoRs are used as the comparison baseline, classifier error directly undermines the verification result.
domain assumption Platforms' internal categorization of moderation decisions is compatible with the DSA-TDB's predefined categories.
The Verification Process compares reconstructed SoRs with original SoRs. If platforms use internal labels that do not map cleanly to DSA fields, matching becomes ambiguous. The paper does not discuss this mapping issue.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving Regulatory Oversight in Online Content Moderation." pith.science (2026). https://pith.science/paper/V5TEWV2D

@misc{pith2026250604145,
  author       = {Pith},
  title        = {Pith review of: Improving Regulatory Oversight in Online Content Moderation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V5TEWV2D}},
  note         = {Machine review of arXiv:2506.04145}
}

read the original abstract

The European Union introduced the Digital Services Act (DSA) to address the risks associated with digital platforms and promote a safer online environment. However, despite the potential of components such as the Transparency Database, Transparency Reports, and Article 40 of the DSA to improve platform transparency, significant challenges remain. These include data inconsistencies and a lack of detailed information, which hinder transparency in content moderation practices. Additionally, the absence of standardized reporting structures makes cross-platform comparisons and broader analyses difficult. To address these issues, we propose two complementary processes: a Transparency Report Cross-Checking Process and a Verification Process. Their goal is to provide both internal and external validation by detecting possible inconsistencies between self-reported and actual platform data, assessing compliance levels, and ultimately enhancing transparency while improving the overall effectiveness of the DSA in ensuring accountability in content moderation. Additionally, these processes can benefit policymakers by providing more accurate data for decision-making, independent researchers with trustworthy analysis, and platforms by offering a method for self-assessment and improving compliance and reporting practices.

Figures

Figures reproduced from arXiv: 2506.04145 by the authors.

**Figure 1.** Overview of the proposed complementary processes for enhancing transparency and detecting inconsistencies in platforms’ reported moderation action. The consistency check between the SoRs in the DSA-TDB and Transparency Reports verifies coherency, while the check between the SoRs and platform shared data verifies their completeness and trustworthiness. state, the SoRs stored in the DSA-TDB fail to answer simple resea… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 19 canonical work pages

[1]

https://eur-lex.europa.eu/legal- content/EN/TXT/HTML/?uri=CELEX:32022R2065

European Parliament and Council, Regulation on a single market for digital services (digital services act) and amending directive, 2022. https://eur-lex.europa.eu/legal- content/EN/TXT/HTML/?uri=CELEX:32022R2065

work page 2022
[2]

Trujillo, T

A. Trujillo, T. Fagni, S. Cresci, The DSA Transparency Database: Auditing self-reported moderation actions by social media, in: ACM CSCW, 2025

work page 2025
[3]

Kaushal, J

R. Kaushal, J. Van De Kerkhof, C. Goanta, G. Spanakis, A. Iamnitchi, Automated transparency: A legal and empirical analysis of the Digital Services Act Transparency Database, in: ACM FAccT, 2024

work page 2024
[4]

Papaevangelou, F

C. Papaevangelou, F. Votta, Content moderation and platform observability in the Digital Services Act, Tech Policy Press (2024)

work page 2024
[5]

Urman, M

A. Urman, M. Makhortykh, How transparent are transparency reports? Comparative analysis of transparency reporting across online platforms, Telecommunications Policy (2023)

work page 2023
[6]

Jaursch, J

J. Jaursch, J. Ohme, U. Klinger, Enabling research with publicly accessible platform data: Early DSA compliance issues and suggestions for improvement (2024)

work page 2024
[7]

Goanta, S

C. Goanta, S. Zannettou, R. Kaushal, J. van de Kerkhof, T. Bertaglia, T. Annabell, H. Gui, G. Spanakis, A. Iamnitchi, The great data standoff: Researchers vs. platforms under the digital services act, arXiv preprint arXiv:2505.01122 (2025)

arXiv 2025
[8]

G. K. Shahi, B. Tessa, A. Trujillo, S. Cresci, A Year of the DSA Transparency Database: What it (Does Not) Reveal About Platform Moderation During the 2024 European Parliament Election, in: ICWSM Workshops, 2025

work page 2024

Show all 20 references

[9]

Dergacheva, V

D. Dergacheva, V. Kuznetsova, R. Scharlach, C. Katzenbach, One day in content moderation: Ana- lyzing 24h of social media platforms’ content decisions through the DSA Transparency Database, Technical Report, Lab Platform Governance, Media, and Technology (PGMT). Centre for Med...

2023
[10]

Bommasani, K

R. Bommasani, K. Klyman, S. Longpre, B. Xiong, S. Kapoor, N. Maslej, A. Narayanan, P. Liang, Foundation model transparency reports, in: AAAI/ACM AIES, volume 7, 2024, pp. 181–195

2024
[11]

van de Kerkhof, Article 22 Digital Services Act: Building trust with trusted flaggers, Internet Policy Review (2025)

J. van de Kerkhof, Article 22 Digital Services Act: Building trust with trusted flaggers, Internet Policy Review (2025)

2025
[12]

C. P. Drolsbach, N. Pröllochs, Content moderation on social media in the EU: Insights from the DSA Transparency Database, in: ACM WebConf Companion, 2024

2024
[13]

H. Li, H. Gao, C. Wu, M. A. Vasarhelyi, Extracting financial data from unstructured sources: Leveraging large language models, Journal of Information Systems (2024)

2024
[14]

Khraisha, S

Q. Khraisha, S. Put, J. Kappenberg, A. Warraitch, K. Hadfield, Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages, Research Synthesis Methods (2024)

2024
[15]

M. Zhu, J. M. Cole, PDFDataExtractor: A tool for reading scientific text and interpreting metadata from the typeset literature in the portable document format, Journal of Chemical Information and Modeling (2022)

2022
[16]

L. Cima, A. Miaschi, A. Trujillo, M. Avvenuti, F. Dell’Orletta, S. Cresci, Contextualized coun- terspeech: Strategies for adaptation, personalization, and evaluation, in: The 34th ACM Web Conference (The Web Conf. ’25), ACM, 2025

2025
[17]

N. S. Mullah, W. M. N. W. Zainon, Advances in machine learning algorithms for hate speech detection in social media: A review, IEEE Access 9 (2021) 88364–88376

2021
[18]

Abdali, S

S. Abdali, S. Shaham, B. Krishnamachari, Multi-modal misinformation detection: Approaches, challenges and opportunities, ACM Computing Surveys 57 (2024) 1–29

2024
[19]

A. M. Almars, Deepfakes detection techniques using deep learning: A survey, Journal of Computer and Communications 9 (2021) 20–35

2021
[20]

Tabone, K

A. Tabone, K. Camilleri, A. Bonnici, S. Cristina, R. Farrugia, M. Borg, Pornographic content classification using deep-learning, in: ACM Symposium on Document Engineering, 2021

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

https://eur-lex.europa.eu/legal- content/EN/TXT/HTML/?uri=CELEX:32022R2065

European Parliament and Council, Regulation on a single market for digital services (digital services act) and amending directive, 2022. https://eur-lex.europa.eu/legal- content/EN/TXT/HTML/?uri=CELEX:32022R2065

work page 2022

[2] [2]

Trujillo, T

A. Trujillo, T. Fagni, S. Cresci, The DSA Transparency Database: Auditing self-reported moderation actions by social media, in: ACM CSCW, 2025

work page 2025

[3] [3]

Kaushal, J

R. Kaushal, J. Van De Kerkhof, C. Goanta, G. Spanakis, A. Iamnitchi, Automated transparency: A legal and empirical analysis of the Digital Services Act Transparency Database, in: ACM FAccT, 2024

work page 2024

[4] [4]

Papaevangelou, F

C. Papaevangelou, F. Votta, Content moderation and platform observability in the Digital Services Act, Tech Policy Press (2024)

work page 2024

[5] [5]

Urman, M

A. Urman, M. Makhortykh, How transparent are transparency reports? Comparative analysis of transparency reporting across online platforms, Telecommunications Policy (2023)

work page 2023

[6] [6]

Jaursch, J

J. Jaursch, J. Ohme, U. Klinger, Enabling research with publicly accessible platform data: Early DSA compliance issues and suggestions for improvement (2024)

work page 2024

[7] [7]

Goanta, S

C. Goanta, S. Zannettou, R. Kaushal, J. van de Kerkhof, T. Bertaglia, T. Annabell, H. Gui, G. Spanakis, A. Iamnitchi, The great data standoff: Researchers vs. platforms under the digital services act, arXiv preprint arXiv:2505.01122 (2025)

arXiv 2025

[8] [8]

G. K. Shahi, B. Tessa, A. Trujillo, S. Cresci, A Year of the DSA Transparency Database: What it (Does Not) Reveal About Platform Moderation During the 2024 European Parliament Election, in: ICWSM Workshops, 2025

work page 2024