Pith. sign in

REVIEW 3 major objections 4 minor 26 references

Validating and monitoring bibliographic and citation data in OpenCitations collections

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OpenCitations' new validator and monitor aim to ensure high-quality open bibliographic and citation data by checking every incoming table against the OpenCitations Data Model and tracking known issues after publication.

desk verdict A solid, useful infrastructure report on real open-source validation and monitoring tools for OpenCitations; the abstract's "precision" claim overreaches because it is not backed by ground-truth evaluation. read the letter →

arxiv 2504.12195 v1 pith:OEBUBANW submitted 2025-04-16 cs.DL

Ivan Heibi , Silvio Peroni , Elia Rizzetto This is my paper · ORCID
classification cs.DL
keywords bibliographicmetadatacitationdataOpenCitationsvalidationqualitymonitoringOCDMSPARQLpersistentidentifiers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that OpenCitations, an open scholarly infrastructure, can keep its bibliographic and citation data reliable by pairing a pre-ingestion validator with a post-publication monitor. The validator checks incoming META-CSV and CITS-CSV tables against the OpenCitations Data Model using four sequential levels of rules, from table well-formedness to the existence of persistent identifiers in official registries. The monitor runs SPARQL queries against the live OpenCitations Meta and Index endpoints to detect known error patterns on a weekly schedule. Applied to a sample from the Matilda source, the validator reported 2,138 metadata issues and 358 citation issues; applied to OpenCitations Meta, the monitor quantified problems such as 1,388,761 bibliographic resources sharing a persistent identifier. If these tools work as described, OpenCitations can safely accept crowdsourced tables while keeping published data under continuous quality surveillance.

What carries the argument

The central object is the OpenCitations Data Model (OCDM), the set of rules defining correct relationships and properties for entities in OpenCitations Meta and Index, together with its tabular incarnation in META-CSV and CITS-CSV files. The validator's four-level pipeline is the mechanism that carries the argument: wellformedness checks table syntax; ID syntax checks PID format against issuing organizations; ID existence queries official registries such as DOI, ORCID, and PMID; semantics checks cross-field consistency like type-identifier compatibility. The monitor's machinery is a JSON-configurable set of SPARQL queries, each encoding a known error pattern, executed against the collections' endpoints and reported with pass/fail status and running time.

What would settle it

Run the validator on a sample of metadata whose identifiers are known to be valid and recently registered; count how many receive a 'br id existence' warning solely because the registry has not yet indexed them. If the count is materially above zero, the precision reported for the Matilda validation is overstated.

Watch

Extended reading notes

Core claim

The central claim is that a purpose-built validation tool, tailored to the OCDM's tabular format, can identify structural and semantic errors in bibliographic and citation data with enough precision to support automated rejection before ingestion, while a SPARQL-based monitoring tool can detect and quantify recurring quality problems in already published collections. The validation tool organises checks into four levels — wellformedness, ID syntax, ID existence, and semantics — and reports each failure with its exact location, validation level, error type (error vs warning), a machine-readable label, and a human-readable message. On the Matilda sample, the tool found frequent warnings for unregistered bibliographic resource identifiers (849 in metadata, 296 in citations), impossible page intervals (664), malformed page formats (522), and other issues. On OpenCitations Meta, the monitoring queries showed that 1,388,761 bibliographic resources share an identifier with at least one other resource, about 1.1% of the collection, with comparable figures for duplicate agent identifiers, demonstrating that known issues can be quantified and tracked over time.

Load-bearing premise

The validator's third level assumes that official persistent identifier registries such as DOI, ORCID, and PMID are complete, current, and always reachable at validation time, so a registry query failure or a not-yet-registered identifier would be reported as a false error.

Editorial extensions

If this is right

  • Incoming tables from crowdsourced contributors can be validated automatically before ingestion, blocking malformed rows while returning precise, human-readable error reports to submitters.
  • Known data quality problems in OpenCitations Meta and Index can be tracked weekly, so regressions introduced by new ingestions become visible immediately.
  • Quantified error counts, such as the 1.1% of bibliographic resources sharing a persistent identifier, give curators a basis for prioritising manual or automated corrections.
  • The validator can double-check internally generated tables from the Source Preprocess step, catching conversion bugs in OpenCitations' own ingestion software before they reach published data.
  • The validator's non-redundancy and maximum-coverage design lets users fix all detectable issues in one pass, reducing resubmission cycles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The validator's registry-based existence checks could be tuned to treat registry failures as warnings rather than errors, avoiding false alarms when a legitimate PID is too new to be indexed.
  • Because the monitor only sees what is expressible in SPARQL over the endpoints, issues that appear only in dump files or API responses escape detection; a complementary dump-level audit would close that gap.
  • The same validation levels could be adapted to other tabular metadata schemas beyond OCDM, since wellformedness, ID syntax, existence, and semantics are generic quality dimensions.
  • If OpenCitations adopts crowdsourcing, the validator's output format could feed an online submission interface that highlights errors directly in the user's table, reducing friction for non-technical contributors.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper describes two software tools developed for the OpenCitations infrastructure: oc_validator, which checks tabular metadata (META-CSV) and citation (CITS-CSV) submissions against the OpenCitations Data Model across four validation levels (wellformedness, ID syntax, ID existence, and semantics), and oc_monitor, which periodically runs SPARQL queries against OpenCitations Meta and Index to detect pre-defined data-quality issues. The tools are evaluated in two scenarios: validation of a sample of Matilda bibliographic and citation data (Tables 2 and 3) and monitoring of OpenCitations Meta with quantitative counts of affected resources (Table 4). The authors claim that the validation tool 'demonstrated its precision' and that the monitoring tool enabled detection and quantification of recurring problems in the OpenCitations Meta collection.

Significance. If the results are taken at face value, the paper contributes a practical, reproducible quality-assurance infrastructure for open scholarly metadata, with publicly available code, data, and a live weekly monitoring service. The evaluation against the external Matilda dataset avoids the circularity of testing on self-generated data, and the integration with GitHub Actions and a public monitoring page is a concrete operational strength. However, the central claim of 'precision' is not supported by the reported evaluation, because the paper provides no ground-truth comparison; moreover, the most frequent warning type depends on external registry lookups whose false-positive rate is unexamined. The monitoring counts are useful for internal tracking but are only as valid as the author-defined SPARQL patterns. These issues are addressable, but they currently weaken the paper's main claims.

major comments (3)
  1. [Abstract and §4.1, Tables 2-3] The paper claims that the validator 'demonstrated its precision' based on counts of errors and warnings, such as 849 'br id existence' warnings in Matilda metadata and 296 in Matilda citations. In error-detection evaluation, precision is the proportion of flagged issues that are true positives, which cannot be established from raw counts alone. No manual audit, no random-sample verification, and no comparison against a reliable second source is reported for any of the flagged issues. The counts show that the tool detects patterns, but they do not demonstrate that the detections are correct. Please either add a ground-truth evaluation (for example, manual inspection of a random sample of each error type, or reconciliation against the source systems) or revise the precision claims in the abstract and Section 4.1.
  2. [§2.2, Validation level 3 (ID Existence)] The ID Existence check queries official PID registries (DOI, ORCID, PMID) to determine whether an identifier is registered. Registry outages, rate limiting, network failures, or identifiers registered after the Matilda data snapshot will cause the validator to flag valid identifiers as non-existent. The paper does not describe how such failures are handled (for example, retries, caching, or timeouts), nor does Section 6 acknowledge this external dependency, even though 'br id existence' is the most frequent warning in both Tables 2 and 3. This is load-bearing for the precision claim: if a non-negligible share of the 849 metadata warnings and 296 citation warnings are lookup failures rather than genuinely unregistered identifiers, the reported counts overstate the actual data problems. Please specify the failure-handling strategy and, where feasible, estimate the false-positive rate by re-querying or manually checking a sample.
  3. [§4.2, Table 4] The monitoring counts are produced by SPARQL queries that encode the authors' own operational definitions of issues such as 'duplicate br count' and 'br with multiple id values count'. This internal benchmarking is legitimate for tracking known issues, but the paper should state more clearly that the counts are only as valid as those patterns and should provide some verification (for example, manual review of a sample of flagged entities) before claiming that the tool enabled quantification of problems. In addition, the identical counts for 'duplicate br count' and 'duplicate id count' (1,388,761) and for 'br in multiple venues count' and 'br with multiple id values count' (760,011) suggest possible overlap between these issue definitions; the paper would benefit from explaining whether these are distinct issues that happen to coincide or the same underlying data condition counted through different queries.
minor comments (4)
  1. [§2.2, Table 1] The example ORCID for Shotton, '0000-0051-5506-523X', does not match the typical ORCID identifier pattern (four groups of four digits, with the first group being a registry-assigned prefix such as 0000-0001, 0000-0002, or 0000-0003 for currently issued identifiers); please verify and correct this example, as it is used to illustrate identifier handling.
  2. [§2.2, first paragraph] Figures 1 and 2 are introduced after Table 1, and the text says 'Table 1, Figure 2, and Figure 1'; please order the references by first mention (Table 1, Figure 1, Figure 2) or renumber the figures.
  3. [§3.2, footnote 8] The phrase 'more than 4,8 billion triples' uses a comma as the decimal separator; for consistency with the rest of the paper, please write '4.8 billion'.
  4. [§6, Conclusions] The final paragraph states that the evaluations 'support the reliability and applicability of the tools'; given the caveats about ground truth and external registry dependencies, this wording should be tempered or the evaluations should be strengthened accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; the validator and monitor are self-contained tools applied to external data and pre-defined issue patterns.

full rationale

The paper's derivation chain is not circular. The validation tool checks Matilda tabular data against the OpenCitations Data Model, external PID syntax rules, and external registries; the reported warnings are counts produced by those checks, not quantities fitted from Matilda or from the tool's own outputs. The monitoring tool executes pre-existing, hand-written SPARQL queries to detect and count known issue types in OpenCitations Meta; this is operational measurement of previously identified problems rather than a claim that the issues are derived from first principles. The self-citations to the authors' earlier OCDM, ingestion, and preliminary-tool papers describe the infrastructure being extended and do not supply the paper's central results. The abstract's word 'precision' is an evidentiary claim lacking a ground-truth audit and is thus a correctness/validity concern, not a circularity: no equation, rule, or fitted parameter is shown to reduce to the paper's own inputs. Overall, the evaluation is anchored to external data (Matilda) and to external or independently defined specifications, so no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the correctness of the OpenCitations Data Model and table specifications (defined in the authors' own prior work), and on the reliability of live external PID registries during validation. No free parameters or invented entities are introduced.

assumptions (3)
  • domain assumption The OpenCitations Data Model and the table specifications (META-CSV and CITS-CSV) define the correctness criteria for the data.
    The validator checks compliance with these internal specs; if the specs themselves are flawed, the validator's notion of 'error' is flawed.
  • domain assumption The external persistent identifier registries (DOI, ORCID, PMID, etc.) are authoritative, complete, and queryable during validation.
    The ID Existence validation level (Section 2.2, level 3) depends on live queries to these registries; failures can cause false errors.
  • domain assumption The known issues tracked by oc_monitor are the relevant data quality problems for the collections.
    The monitor only detects pre-identified errors; if the error catalog is incomplete, the monitoring is incomplete. See Section 2.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Validating and monitoring bibliographic and citation data in OpenCitations collections." pith.science (2026). https://pith.science/paper/OEBUBANW

@misc{pith2026250412195,
  author       = {Pith},
  title        = {Pith review of: Validating and monitoring bibliographic and citation data in OpenCitations collections},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OEBUBANW}},
  note         = {Machine review of arXiv:2504.12195}
}
read the original abstract

Purpose. The increasing emphasis on data quantity in research infrastructures has highlighted the need for equally robust mechanisms ensuring data quality, particularly in bibliographic and citation datasets. This paper addresses the challenge of maintaining high-quality open research information within OpenCitations, a community-guided Open Science Infrastructure, by introducing tools for validating and monitoring bibliographic metadata and citation data. Methods. We developed a custom validation tool tailored to the OpenCitations Data Model (OCDM), designed to detect and explain ingestion errors from heterogeneous sources, whether due to upstream data inconsistencies or internal software bugs. Additionally, a quality monitoring tool was created to track known data issues post-publication. These tools were applied in two scenarios: (1) validating metadata and citations from Matilda, a potential future source, and (2) monitoring data quality in the existing OpenCitations Meta dataset. Results. The validation tool successfully identified a variety of structural and semantic issues in the Matilda dataset, demonstrating its precision. The monitoring tool enabled the detection of recurring problems in the OpenCitations Meta collection, as well as their quantification. Together, these tools proved effective in enhancing the reliability of OpenCitations' published data. Conclusion. The presented validation and monitoring tools represent a step toward ensuring high-quality bibliographic data in open research infrastructures, though they are limited to the data model adopted by OpenCitations. Future developments are aimed at expanding to additional data sources, with particular regard to crowdsourced data.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 18 canonical work pages

  1. [27]

    In: Proceedings of the 23rd International Conference on World Wide Web, pp

    Kontokostas, D., Westphal, P., Auer, S., Hellmann, S., Lehmann, J., Cornelissen, R., Zaveri, A.: Test-driven evaluation of linked data quality. In: Proceedings of the 23rd International Conference on World Wide Web, pp. 747–758. ACM, Seoul Korea (2014). https://doi.org/10.1145/2566486.2568002 23

    arXiv 2014

  2. [1]

    Mobile Networks and Applica- tions 19(2), 171–209 (2014) https://doi.org/10.1007/s11036-013-0489-0

    Chen, M., Mao, S., Liu, Y.: Big Data: A Survey. Mobile Networks and Applica- tions 19(2), 171–209 (2014) https://doi.org/10.1007/s11036-013-0489-0

    doi:10.1007/s11036-013-0489-0 2014

  3. [2]

    European Science Foundation (2022)

    Coalition for Advancing Research Assessment: Agreement on Reforming Research Assessment. European Science Foundation (2022). https://coara.eu/agreement/ the-agreement-full-text/

    work page 2022

  4. [3]

    Barcelona Declaration on Open Research Information, Kramer, B., Neylon, C., Waltman, L.: Barcelona Declaration on Open Research Information (2024) https: //doi.org/10.5281/ZENODO.10958522

    work page doi:10.5281/zenodo.10958522 2024

  5. [4]

    Nature 520(7548), 429–431 (2015) https://doi.org/10.1038/520429a

    Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., Rafols, I.: Bibliometrics: The Leiden Manifesto for research metrics. Nature 520(7548), 429–431 (2015) https://doi.org/10.1038/520429a

    doi:10.1038/520429a 2015

  6. [5]

    Scientific Data 3(1), 160018 (2016) https://doi.org/10.1038/sdata.2016.18

    Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., Da Silva Santos, L.B., Bourne, P.E., Bouw- man, J., Brookes, A.J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C.T., Finkers, R., Gonzalez-Beltran, A., Gray, A.J.G., Groth, P., Goble, C., Grethe, J.S., Heringa, J., ’T Hoe...

    2016

  7. [6]

    Manola, N., Tzouganatou, A., Kuchma, I., CoARA WG on OI4RRA: Open Infras- tructures for Responsible Research Assessment: Principles and Framework (2025) https://doi.org/10.5281/ZENODO.14844582

    work page doi:10.5281/zenodo.14844582 2025

  8. [7]

    Bilder, G., Lin, J., Neylon, C.: The Principles of Open Scholarly Infrastructure (2020) https://doi.org/10.24343/C34W2H

    doi:10.24343/c34w2h 2020

Show all 26 references
  1. [8]

    Scientometrics (2024) https: //doi.org/10.1007/s11192-024-05160-7 21

    Heibi, I., Moretti, A., Peroni, S., Soricetti, M.: The OpenCitations Index: Descrip- tion of a database providing open citation data. Scientometrics (2024) https: //doi.org/10.1007/s11192-024-05160-7 21

    2024 doi

  2. [10]

    Quantitative Science Studies1(1), 414– 427 (2020) https://doi.org/10.1162/qss a 00022

    Hendricks, G., Tkaczyk, D., Lin, J., Feeney, P.: Crossref: The sustainable source of community-owned scholarly metadata. Quantitative Science Studies1(1), 414– 427 (2020) https://doi.org/10.1162/qss a 00022

    2020 doi

  3. [11]

    DataCite Metadata Working Group: DataCite Metadata Schema Documentation for the Publication and Citation of Research Data and Other Research Outputs v4.5 (2024) https://doi.org/10.14454/G8E5-6293

    2024 doi

  4. [12]

    PLOS Biology 17(10), 3000385 (2019) https://doi.org/10.1371/journal.pbio.3000385

    Hutchins, B.I., Baker, K.L., Davis, M.T., Diwersy, M.A., Haque, E., Harriman, R.M., Hoppe, T.A., Leicht, S.A., Meyer, P., Santangelo, G.M.: The NIH Open Citation Collection: A public access, broad coverage resource. PLOS Biology 17(10), 3000385 (2019) https://doi.org/10.1371/j...

    2019 doi

  5. [13]

    In: Grana, C., Baraldi, L

    Atzori, C., Bardi, A., Manghi, P., Mannocci, A.: The OpenAIRE Workflows for Data Management. In: Grana, C., Baraldi, L. (eds.) Digital Libraries and Archives vol. 733, pp. 95–107. Springer, Cham (2017). https://doi.org/10.1007/ 978-3-319-68130-6 8

    2017

  6. [14]

    D-Lib Magazine 18(9/10) (2012) https://doi.org/10.1045/september2012-manghi

    Manghi, P., Bolikowski, L., Manold, N., Schirrwagen, J., Smith, T.: Ope- nAIREplus: The European Scholarly Communication Data Infrastructure. D-Lib Magazine 18(9/10) (2012) https://doi.org/10.1045/september2012-manghi

    2012 doi

  7. [15]

    Journal of Information Processing and Management 55(1), 42–46 (2012) https: //doi.org/10.1241/johokanri.55.42

    Kato, T., Tsuchiya, E., Kubota, S., Miyagawa, Y.: Japan Link Center (JaLC): Link management and DOI assignment for Japanese electronic scholarly contents. Journal of Information Processing and Management 55(1), 42–46 (2012) https: //doi.org/10.1241/johokanri.55.42

    2012 doi

  8. [16]

    In: Pan, J.Z., Tamma, V., d’Amato, C., Janowicz, K., Fu, B., Polleres, A., Senevi- ratne, O., Kagal, L

    Daquino, M., Peroni, S., Shotton, D., Colavizza, G., Ghavimi, B., Lauscher, A., Mayr, P., Romanello, M., Zumstein, P.: The OpenCitations Data Model. In: Pan, J.Z., Tamma, V., d’Amato, C., Janowicz, K., Fu, B., Polleres, A., Senevi- ratne, O., Kagal, L. (eds.) The Semantic Web ...

    2020

  9. [17]

    Journal of Open Humanities Data 10, 27 (2024) https://doi.org/10.5334/ johd.190

    Mal´ ınek, V., Umerle, T., Gray, E., Heibi, I., Kir´ aly, P., Klaes, C., Korytkowski, P., Lindemann, D., Moretti, A., Panuˇ skov´ a, C., P´ eter, R., Tolonen, M., Tomczy´ nska, A., Vimr, O.: Open Bibliographical Data Workflows and the Multilinguality Chal- lenge. Journal of Op...

    2024

  10. [18]

    arXiv (2024)

    Massari, A., Peroni, S.: HERITRACE: Tracing Evolution and Bridging Data for Streamlined Curatorial Work in the GLAM Domain. arXiv (2024). https://doi. org/10.48550/ARXIV.2402.00477 22

    2024 doi

  11. [19]

    In: Cor- nia, M., Nunzio, G.M.D., Firmani, D., Mizzaro, S., Serra, G., Tonelli, S., Trema- munno, A

    Peroni, S., Rizzetto, E.: A Tool for Validating and Monitoring Bibliographic Data in Open Research Information Systems: The OpenCitations Collections. In: Cor- nia, M., Nunzio, G.M.D., Firmani, D., Mizzaro, S., Serra, G., Tonelli, S., Trema- munno, A. (eds.) Proceedings of the...

    2025

  12. [20]

    In: ELPUB 2019 23d International Confer- ence on Electronic Publishing

    Torny, D., Capelli, L., Danjean, L.: Matilda: Building a bibliographic/metric tool for open citations and open science. In: ELPUB 2019 23d International Confer- ence on Electronic Publishing. OpenEdition Press, ??? (2019). https://doi.org/ 10.4000/proceedings.elpub.2019.22

    2019 doi

  13. [21]

    OpenCitations (2024)

    Moretti, A., Massari, A., Rizzetto, E., Soricetti, M., Heibi, I.: Oc ds converter. OpenCitations (2024). https://doi.org/10.5281/zenodo.12911527

    2024 doi

  14. [22]

    arXiv (2019)

    Heibi, I., Peroni, S., Shotton, D.: Crowdsourcing Open Citations with CROCI – An Analysis of the Current Status of Open Citations, and a Proposal. arXiv (2019). https://doi.org/10.48550/arXiv.1902.02534

    2019 doi arXiv

  15. [23]

    arXiv (2022)

    Massari, A., Heibi, I.: How to Structure Citations Data and Bibliographic Meta- data in the OpenCitations Accepted Format. arXiv (2022). https://doi.org/10. 48550/arXiv.2206.03971

    2022 doi arXiv

  16. [24]

    Zenodo (2022)

    Massari, A.: How to Produce Well-Formed CSV Files for OpenCitations. Zenodo (2022). https://doi.org/10.5281/zenodo.6597141

    2022 doi

  17. [25]

    Zenodo (2025)

    Rizzetto, E., Heibi, I.: Material for the Evaluation of Oc validator with Matilda and of Oc monitor with OpenCitations Meta. Zenodo (2025). https://doi.org/10. 5281/ZENODO.15224594

    2025

  18. [26]

    Semantic Web 7(1), 63–93 (2015) https://doi.org/10.3233/ SW-150175

    Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., Auer, S.: Quality assessment for Linked Data: A Survey: A systematic literature review and con- ceptual framework. Semantic Web 7(1), 63–93 (2015) https://doi.org/10.3233/ SW-150175

    2015

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.