REVIEW 4 major objections 4 minor 44 references
From Reports to Reality: Testing Consistency in Instagram's Digital Services Act Compliance Data
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Instagram's DSA filings disagree with each other by millions of moderation actions.
desk verdict Useful framework and real internal-consistency findings, but the external-consistency centerpiece (Table 3) compares non-aligned periods and categories, so the 'compliance failures' inference doesn't hold; fix that and this is a solid case study. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The framework's core machinery is a three-level consistency grid. Level 1 (internal) checks that numbers within one report reconcile, e.g., that sums of policy categories equal reported totals. Level 2 (external) cross-checks the same quantity across mechanisms, e.g., transparency-report 'account terminations' against SOR statements of reason. Level 3 (historical) compares figures across reporting periods and flags unexplained category reshuffling. The discrepancy itself is the unit of analysis: the framework converts public data from four separate instruments into a single, triangulated indicator of where reporting cannot be reconciled.
What would settle it
If a platform operator published a decision-level mapping that links every transparency-report termination total to SOR statements with matching timestamps and reasons, and the reconciled counts agreed to within a small rounding tolerance for Reports 2–4, the paper's central inconsistency claim would be falsified for those periods.
Extended reading notes
Core claim
On its own terms, the paper establishes that Instagram's reporting across four DSA-mandated mechanisms is inconsistent in ways that map onto a three-level taxonomy. Internal checks find that reported totals do not equal the sum of disaggregated categories, with deviations up to 71.4 million in the first report. External checks find that account terminations reported in transparency reports differ from SOR records by more than 30 percent in Reports 2–3 and reverse sign in Report 4, where SOR records exceed the transparency report by 5.26 million. Historical checks find that total removals fell by more than half, then by two-thirds, then by 44 percent across the four reporting periods, with redefined categories such as the introduction of 'Spam' accounting for most of the volume in Report 2 only to collapse in Report 3. The paper reads these patterns as evidence of fragmented data pipelines and category misalignment rather than as direct proof of under-reporting, and it treats the platform's own risk assessment and the independent audit — which acknowledge post-hoc SOR generation and asynchronous logging — as corroborating the interpretation.
Load-bearing premise
The external-consistency comparison assumes that the account-termination figures in transparency reports and the statement-of-reason records in the SOR database refer to the same set of moderation actions and that the SOR records are complete enough to serve as a benchmark, an assumption the platform's own risk assessment and the independent audit both qualify.
Editorial extensions
If this is right
- Regulators can treat unexplained cross-mechanism gaps, such as the 5.26 million reverse gap in Report 4, as triggers for formal inquiry under DSA Articles 34 and 35 rather than waiting for isolated complaints.
- Independent audits under Article 37 should include quantitative reconciliation of transparency reports with SOR records as a standard control step.
- If consistency is a compliance diagnostic, then the Delegated Regulation's mandatory templates and keyword mappings, once in force, should make such checks automatable and comparable across very large online platforms.
- Evaluating platforms as an interconnected reporting ecosystem supersedes the earlier paradigm of judging transparency reports in isolation, since inconsistencies only appear when mechanisms are triangulated.
- Sharp historical swings that are not explained in report narratives, such as the 67 percent drop between Reports 2 and 3, are themselves reporting deficiencies under the DSA's requirement for meaningful and comprehensible information.
Reading between the lines
- The authors leave implicit that if definitional mismatches explain most of the external gap, then the DSA Transparency Database's SOR data are not yet usable as ground truth for research or enforcement, which would limit the database's value until the Commission enforces reconciliation.
- The same three-level test could be run cross-platform: ranking platforms by the size and direction of transparency-report/SOR gaps would separate systemic category-design problems from platform-specific data-handling failures.
- The sharpest testable extension is decision-level tracing: sampling moderation decisions from internal logs and following them through SOR entries and transparency-report totals would determine whether the Report 4 reverse gap comes from SOR statements that never reach transparency-report terminations, which would itself be a compliance-relevant discrepancy.
- Re-running the identical consistency checks on post-July-2025 reports, after the Delegated Regulation's templates become mandatory, would test whether formal harmonization actually shrinks the cross-mechanism gaps the paper documents.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a three-level consistency framework (internal, external, historical) for evaluating platform compliance under the EU Digital Services Act (DSA), and applies it to Instagram's first four DSA transparency reports, the Statement of Reasons (SOR) database, the 2024 systemic risk assessment (SRA), and the EY independent audit. It documents internal arithmetic mismatches in transparency report tables, term and category inconsistencies across reporting instruments, and qualitative admissions by Meta and EY about fragmented reporting systems. The paper concludes that Instagram's reporting is inconsistent across these mechanisms and argues that such consistency checks 'can help detect potential compliance failures.' The central empirical support for this inference is an external-consistency comparison (Table 3) between transparency-report 'account terminations' and SOR-recorded 'restrictions.'
Significance. If the core empirical comparison were valid, the paper would make a useful contribution by triangulating four DSA-mandated transparency mechanisms and showing how researchers and regulators could use cross-instrument checks. The paper's strengths include its use of public primary sources (transparency reports, SOR database, SRA, and audit), its explicit citation of Meta's SRA admission that some SOR datasets are 'generated post hoc from logging systems' and EY's finding of 'asynchronous logging systems,' and its documentation of concrete internal arithmetic deviations in Table 1. These qualitative findings and internal inconsistencies are credible and support a weaker conclusion about fragmented reporting and the need for harmonization. However, the strongest claim—that consistency checks can detect potential compliance failures—rests on Table 3, which compares non-equivalent categories over non-overlapping periods and treats known pipeline artifacts as potential misreporting. The paper's own evidence undercuts this inference, so the novel quantitative external-consistency test does not currently support the headline claim.
major comments (4)
- [§4.2, Table 3] The external-consistency comparison in Table 3 is not apples-to-apples. Section 5.1 acknowledges that Instagram's transparency reports use 'account terminations' while SOR entries use 'account restrictions,' and the paper does not demonstrate that the SOR filter used for Table 3 selects exactly the decisions reported in transparency-report Table 15.1.c.(3)/(4). Under a definitional mismatch, the reported differences—including the negative difference of -5,259,075 for Report 4—are expected even under fully compliant reporting. The table therefore does not support the inference that the gaps indicate potential compliance failures.
- [§8.1, Table 3] Section 8.1 concedes that Report 1 predates the operational launch of the SOR database on 26 September 2023. Yet Table 3 includes a TR1-SOR difference of 9,396,627 as if it were a meaningful discrepancy. This difference is a pure temporal artifact because no SOR entries could exist for that period; including it systematically inflates the evidence for external inconsistency and should either be removed or explicitly labeled as non-comparable.
- [§4.2, §8] The paper's own cited evidence undermines the use of SOR counts as a benchmark for external consistency. Meta's SRA states that some SOR datasets 'are generated post hoc from logging systems rather than sourced directly from moderation workflows,' and EY's audit identifies 'asynchronous logging systems' as a barrier to validating SOR completeness (Section 4.2). These conditions imply that SOR counts will be systematically lower than transparency-report totals even when reporting is accurate. The paper acknowledges these limitations but does not adjust its inference; the observed gaps therefore cannot distinguish misreporting from known pipeline and definitional mismatches, directly weakening the abstract's claim that consistency 'can help detect potential compliance failures.'
- [§4, §5] The proposed three-level consistency framework is not operationalized into testable criteria. Internal consistency is checked by summing table columns, external consistency is assessed through a single unadjusted comparison in Table 3, and historical consistency (Section 4.3) interprets category renames and merges as inconsistencies without analyzing whether the underlying definitions changed across reports. Without explicit decision rules for what counts as a material inconsistency, the framework functions as a heuristic rather than a validated methodology, and the paper's claim of a 'multi-level consistency framework' as a contribution needs either more precise operationalization or more cautious framing.
minor comments (4)
- [§2, References [7] and [11]] The citation for the SOR database is inconsistent: Section 2 cites reference [7] for 'over 32 million moderation decisions,' but [7] is a European Commission press release about the Terms and Conditions database, not the DSA Transparency Database. Please correct the reference to the SOR/Transparency Database source.
- [§8, 'Summary of Contributions'] The summary states that the paper assesses 'four levels of consistency—internal, external, historical, and cross-mechanism,' but Sections 2 and 4 define only three levels (internal, external, historical) and treat cross-mechanism comparison as part of external consistency. Please reconcile the terminology.
- [§2.1] There is a typo: 'the first issed round of audit reports' should read 'the first issued round of audit reports.'
- [Table 1] The final row of Table 1 lacks thousands separators (e.g., '35,724,61312,136,947'), making the numbers difficult to read. Please format all entries consistently.
Circularity Check
No significant circularity: the consistency framework is an analytic lens applied to independent public datasets, with no fitted parameters or self-citation chain driving the conclusions.
full rationale
The paper does not derive its conclusions from self-defined constructs in a way that collapses into its inputs. It proposes a three-level consistency framework (internal, external, historical) and applies it to independently published DSA mechanisms: Meta's transparency reports, the European Commission's SOR database, Meta's systemic risk assessment, and EY's independent audit. No parameter is fitted and no quantity is predicted from a fitted value; the observed gaps are arithmetic comparisons and qualitative cross-document alignments. The author self-references ([36], [37], [43]) support contextual claims about NetzDG reporting and audit sampling and are not load-bearing for the Instagram/SOR comparison. The paper explicitly disclaims definitive findings ('They do not represent a definitive compliance audit', Section 8.1) and concedes the main threats to the external-consistency test: terminology mismatches ('account terminations' vs 'account restrictions'), non-overlapping periods for Report 1, and post-hoc or asynchronous SOR logging, quoting Meta's SRA and EY's audit. Those concessions weaken the compliance-failure inference, but that is a validity or correctness concern, not circularity. The consistency framework is an analytic lens rather than a self-justifying derivation, so no circular step can be exhibited.
Assumptions & free parameters
assumptions (4)
- domain assumption Consistency across reporting mechanisms is a valid indicator of compliance or misreporting.
- domain assumption Transparency report totals should equal the sum of category subtotals, per Annex II of the Delegated Regulation.
- domain assumption The SOR database can serve as a comparative benchmark for moderation decisions attributable to Instagram.
- domain assumption Reporting periods across mechanisms overlap sufficiently for comparison.
Cite this review
Pith. "Pith review of From Reports to Reality: Testing Consistency in Instagram's Digital Services Act Compliance Data." pith.science (2026). https://pith.science/paper/5CQ3A2MI
@misc{pith2026250701787,
author = {Pith},
title = {Pith review of: From Reports to Reality: Testing Consistency in Instagram's Digital Services Act Compliance Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/5CQ3A2MI}},
note = {Machine review of arXiv:2507.01787}
}
read the original abstract
The Digital Services Act (DSA) introduces harmonized rules for content moderation and platform governance in the European Union, mandating robust compliance mechanisms, particularly for very large online platforms and search engines. This study examined compliance with DSA requirements, focusing on Instagram as a case study. We develop and apply a multi-level consistency framework to evaluate DSA compliance. Our findings contribute to the broader discussion on empirically-based regulation, providing insight into how researchers, regulators, auditors and platforms can better utilize DSA mechanisms to improve reporting and enforcement quality and accountability. This work underscores that consistency can help detect potential compliance failures. It also demonstrates that platforms should be evaluated as part of an interconnected ecosystem rather than through isolated processes, which is crucial for effective compliance evaluation under the DSA.
Figures
Reference graph
Works this paper leans on
-
[1]
Roberts, CDT, Open Technology Institute, Nicolas Suzor, and Sarah Myers West
ACLU, EFF, Irina Raicu, Sarah T. Roberts, CDT, Open Technology Institute, Nicolas Suzor, and Sarah Myers West. 2018. Santa Clara Principles on Transparency and Accountability in Content Moderation. https://santaclaraprinciples.org/images/scp-og.png
work page 2018
-
[2]
BfJ Press agency. 2019. Federal Office of Justice Issues Fine against Facebook. https://www.bundesjustizamt.de/DE/Presse/Archiv/2019/20190702_ EN.html;jsessionid=2991FA58C1ACD667E932F1E75AC007FB.2_cid370?nn=3451904
work page 2019
-
[3]
Austrian Federal Government. 2020. Bundesgesetz über Maßnahmen zum Schutz der Nutzer auf Kommunikationsplattformen (Kommunikationsplattformen-Gesetz – KoPl-G – Communication Platforms Act). https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage= Bundesnormen&Gesetzesnummer=20011415
work page 2020
-
[4]
Reuben Binns. 2018. Fairness in Machine Learning: Lessons from Political Philosophy. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81) , Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, Stockholm, Sweden, 149–159. https://proceedings.mlr.press/v81/binns18a.html Manuscr...
work page 2018
-
[5]
Jim Buatti, Lead Counsel, and Wikimedia Foundationand Aeryn Palmer. 2017. Wikimedia Foundation releases first transparency report of 2017. https://diff.wikimedia.org/2017/02/13/sixth-transparency-report/
work page 2017
-
[6]
Commission. 2023. Statements of Reasons, DSA Transparency Database. https://transparency.dsa.ec.europa.eu/statement
work page 2023
-
[8]
Chiara Drolsbach and Nicolas Pröllochs. 2023. Content Moderation on Social Media in the EU: Insights From the DSA Transparency Database. http://arxiv.org/abs/2312.04431 arXiv:2312.04431 [cs]
arXiv 2023
-
[9]
European Commission. 2022. Regulation (EU) 2022/2065 of the European Parliament and of the Council of 19 October 2022 on a Single Market For Digital Services and amending Directive 2000/31/EC (Digital Services Act) (Text with EEA relevance). http://data.europa.eu/eli/reg/2022/2065/oj/eng Legislative Body: EP, CONSIL
work page 2022
Show all 44 references
-
[10]
European Commission. 2023. Commission Delegated Regulation (EU) .../... of 20 October 2023 supplementing Regulation (EU) 2022/2065 of the European Parliament and of the Council, by laying down rules on the performance of audits for very large online platforms and very large on...
2023
-
[11]
European Commission. 2023. Commission launches new database to track digital services terms and conditions | Shaping Europe’s digital future. https://digital-strategy.ec.europa.eu/en/news/commission-launches-new-database-track-digital-services-terms-and-conditions
2023
-
[12]
European Commission. 2024. Annex II of the Delegated Regulation for DSA Transparency Reporting (Implementing Regulation laying down templates concerning the transparency reporting obligations of providers of online platforms ). https://digital-strategy.ec.europa.eu/en/library/...
2024
-
[13]
European Commission. 2024. Commission Implementing Regulation (EU) .../... of 4 November 2024 laying down templates concerning the transparency reporting obligations of providers of intermediary services and of providers of online platforms under Regulation (EU) 2022/2065 of t...
2024
-
[14]
European Commission. 2024. Commission opens formal proceedings under DSA. https://ec.europa.eu/commission/presscorner/detail/en/IP_24_2373
2024
-
[15]
European Commission. 2024. Digital Services Act / Transparency Database / dsa-tdb· GitLab. https://code.europa.eu/dsa/transparency-database/dsa- tdb
2024
-
[16]
European Commission. 2025. Digital Services Act: Commission concludes that TikTok breached obligations on advertising transparency. https: //ec.europa.eu/commission/presscorner/detail/en/ip_25_1223. Accessed: 20 May 2025
2025
-
[17]
German Federal Ministry of Justice. 2017. German Network Enforcement Act (NetzDG). https://www.bmj.de/DE/Themen/FokusThemen/NetzDG/ NetzDG_EN_node.html
2017
-
[18]
Amélie Heldt. 2019. Reading between the lines and the numbers: an analysis of the first NetzDG reports. Internet Policy Review 8, 2 (June 2019), 1–19. https://policyreview.info/articles/analysis/reading-between-lines-and-numbers-analysis-first-netzdg-reports
2019
- [19]
-
[20]
Daphne Keller. 2022. The DSA’s Industrial Model for Content Moderation . Verfassungsblog. https://verfassungsblog.de/dsa-industrial-model/
2022
-
[21]
Daphne Keller. 2023. The Rise of the Compliant Speech Platform . Lawfare. https://www.lawfaremedia.org/article/the-rise-of-the-compliant-speech- platform
2023
-
[22]
Thomas F. Keller. 1968. Discussion of The Auditing Standard of Consistency. Journal of Accounting Research 6 (1968), 18–22. doi:10.2307/2490064 Publisher: [Accounting Research Center, Booth School of Business, University of Chicago, Wiley]
1968 doi
-
[23]
Vlasis Koutsos, Xiangan Tian, Dimitrios Papadopoulos, and Dimitris Chatzopoulos. 2024. Cross Ledger Transaction Consistency for Financial Auditing. In 6th Conference on Advances in Financial Technologies (AFT 2024) (Leibniz International Proceedings in Informatics (LIPIcs), Vo...
2024 doi
-
[24]
Kroll, Joanna Huey, Solon Barocas, Edward W
Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable Algorithms. University of Pennsylvania Law Review 165, 3 (2017), 633–705. https://scholarship.law.upenn.edu/penn_law_review/vol165/iss3/3/
2017
-
[25]
Bethan McKernan. 2023. Israel and Hamas at war after surprise attacks from Gaza Strip. https://www.theguardian.com/world/2023/oct/07/hamas- launches-surprise-attack-on-israel-as-palestinian-gunmen-reported-in-south World news
2023
-
[26]
Meta. 2024. Instagram Transparency Report April-September. Technical Report. Meta. https://scontent-ams4-1.xx.fbcdn.net/v/t39.8562-6/466943155_ 1291701138400105_7867447844898917200_n.pdf?_nc_cat=103&ccb=1-7&_nc_sid=b8d81d&_nc_ohc=a77fZ3TJmhEQ7kNvgGXTNZg&_nc_zt=14& _nc_ht=scont...
2024
-
[27]
Meta. 2024. Transparency Report for Instagram . Technical Report. Meta. https://scontent-ams4-1.xx.fbcdn.net/v/t39.8562-6/446849908_ 1440652056655341_3316525698930778970_n.pdf?_nc_cat=103&ccb=1-7&_nc_sid=b8d81d&_nc_ohc=kiNdYQXaOUQQ7kNvgFH8Sc0&_nc_ht= scontent-ams4-1.xx&oh=00_A...
2024
-
[28]
Meta Platforms Ireland Limited. 2023. Regulation (EU) 2022/2065 Digital Services Act Transparency Report for Instagram: First Report. https://scontent-ams2-1.xx.fbcdn.net/v/t39.8562-6/447971060_1481740992549061_1404827436118992020_n.pdf?_nc_cat=100&ccb=1-7&_nc_ sid=b8d81d&_nc_...
2023
-
[29]
Meta Platforms Ireland Limited. 2024. Instagram Systemic Risk Assessment under the Digital Services Act. https://about.instagram.com/dsa/systemic- risk-assessment-2024 Accessed May 2025
2024
-
[30]
Meta Platforms Ireland Limited. 2024. Regulation (EU) 2022/2065 Digital Services Act Transparency Report for Instagram: Second Report. https://scontent-ams2-1.xx.fbcdn.net/v/t39.8562-6/448110749_316590818169033_3098716353127960367_n.pdf?_nc_cat=111&ccb=1-7&_nc_ sid=b8d81d&_nc_...
2024
-
[31]
Meta Platforms Ireland Limited. 2024. Regulation (EU) 2022/2065 Digital Services Act Transparency Report for Instagram: Third Re- port. https://scontent-ams4-1.xx.fbcdn.net/v/t39.8562-6/466943155_1291701138400105_7867447844898917200_n.pdf?_nc_cat=103&ccb=1- 7&_nc_sid=b8d81d&_n...
2024
-
[32]
Meta Platforms Ireland Limited. 2025. Regulation (EU) 2022/2065 Digital Services Act Transparency Report for Instagram: Fourth Report. https://scontent-ams4-1.xx.fbcdn.net/v/t39.8562-6/492672947_1857060258361688_ 2578453831459945466_n.pdf?_nc_cat=102&ccb=1-7&_nc_sid=b8d81d&_nc...
2025
-
[33]
OECD. 2020. Current approaches to terrorist and violent extremist content among the global top 50 online content-sharing services . Technical Report. OECD, Paris. doi:10.1787/68058b95-en
2020 doi
-
[34]
Christopher Parsons. 2019. The (In)effectiveness of Voluntarily Produced Transparency Reports. Business & Society 58, 1 (2019), 103–131. doi:10.1177/0007650317717957
2019 doi
-
[35]
Reuters. 2019. Germany fines Facebook for under-reporting complaints. https://www.reuters.com/article/us-facebook-germany-fine- idUSKCN1TX1IC
2019
- [36]
-
[37]
It Is Unfair, And It Would Be Unwise To Expect The User To Know The Law!
Marie-Therese Sekwenz, Ben Wagner, and Simon Parkin. 2025. “It Is Unfair, And It Would Be Unwise To Expect The User To Know The Law!” – Evaluating Reporting Mechanisms Under The Digital Services Act. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Tr...
2025
- [38]
-
[39]
TikTok. 2023. DSA Transparency report . Technical Report 1. TikTok. https://sf16-va.tiktokcdn.com/obj/eden-va2/zayvwlY_ fjulyhwzuhy[/ljhwZthlaukjlkulzlp/DSA_H2_2024/TikTok-DSA-Transparency-Report-Jan-to-Jun-2024.pdf
2023
-
[40]
TikTok. 2024. DSA Transparency report. Technical Report 2. TikTok. https://www.tiktok.com/transparency/en/dsa-transparency/
2024
-
[41]
TikTok. 2024. TikTok DSA Transparency Report Jan to Jun 2024 . Technical Report. TikTok. https://sf16-va.tiktokcdn.com/obj/eden-va2/zayvwlY_ fjulyhwzuhy[/ljhwZthlaukjlkulzlp/DSA_H2_2024/TikTok-DSA-Transparency-Report-Jan-to-Jun-2024.pdf
2024
- [42]
-
[43]
Ben Wagner, Krisztina Rozgonyi, Marie-Therese Sekwenz, Jennifer Cobbe, and Jatinder Singh. 2020. Regulating transparency? Facebook, Twitter and the German Network Enforcement Act. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20) . ...
2020
-
[44]
Ernst & Young and Meta Platforms Ireland Limited. 2024. Instagram Independent Audit Report: Digital Services Act Systemic Risk As- sessment 2024 . Technical Report. Meta Platforms Ireland Limited. https://scontent-ams4-1.xx.fbcdn.net/v/t39.8562-6/468562318_ 2548652918858554_79...
2024
-
[45]
Rebecca Zipursky. 2019. Nuts About NETZ: The Network Enforcement Act and Freedom of Expression. Fordham International Law Journal 42, 4 (2019), 1325. https://ir.lawnet.fordham.edu/ilj/vol42/iss4/7 Manuscript submitted to ACM From Reports to Reality 19 10 Appendices A Overview ...
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.