{"id":"0b88e16c-d283-4ba0-b054-671bf2e09945","arxiv_id":"2504.12195","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces oc_validator and oc_monitor, two open-source tools that check bibliographic metadata and citation data for compliance with the OpenCitations Data Model and track known quality issues.","lead":"OpenCitations researchers built two open-source tools: one validates incoming metadata and citation tables against the OpenCitations Data Model, and one monitors known data quality issues in the published collections. The validation tool was tested on data from the Matilda bibliometric tool, and the monitoring tool quantified recurring errors in OpenCitations Meta.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Precision claim is unsupported because no ground truth exists for the 'br id existence' warnings","rationale":"The reader's weakest assumption identified the same load-bearing concern: the ID Existence validation level (Section 2.2, level 3) assumes official PID registries are complete, up-to-date, and reliably queryable. I agree this is the most load-bearing point, and I refine it by tying it directly to the paper's headline claim of 'demonstrating its precision' and to the raw counts in Tables 2 and 3. If the external registry lookups are unreliable or time-sensitive, the validator's reported 'br id existence' counts are not necessarily true positives, so the precision claim is unsupported. I recommend keeping the reader's CONDITIONAL verdict: the tools exist and demonstrably run their checks, but the evaluative claim of precision requires a ground-truth audit. No other concern (e.g., the absence of commit hashes or the lack of a baseline comparison) is as directly load-bearing for the central claim. The paper is otherwise a sound methods-and-tools paper with public code and reproducible materials, and I credit that; the conditional recommendation reflects the need for one additional audit step rather than a rejection.","tokens_in":15009,"tokens_out":1620,"duration_ms":14027,"concrete_test":"Take a random sample of, say, 100 of the 849 'br id existence' warnings from the Matilda metadata report and manually verify each identifier as follows: (1) query the relevant registry (DOI via DataCite/CrossRef, PMID via NCBI, ORCID via the ORCID API) at a later date with multiple retries; (2) check whether the identifier is registered but the original automated lookup failed due to a transient error or rate limit; (3) check whether the identifier was registered after the September 2024 Matilda snapshot. If more than a small threshold (e.g., 5%) of the sampled warnings are false positives (identifiers that are in fact registered), the 'precision' claim and the raw warning counts in Tables 2 and 3 need to be reinterpreted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that oc_validator demonstrates 'precision' because it identified 849 'br id existence' warnings in Matilda metadata and 296 in Matilda citations (Tables 2 and 3). But 'precision' in any error-detection sense requires distinguishing true positives from false positives. No ground truth is established for these warnings: the validator treats an identifier as non-existent when a query to an external registry (DOI, ORCID, PMID) fails or returns no registration. Registry outages, rate limits, network failures, or valid identifiers registered after the Matilda snapshot date would produce false 'br id existence' warnings. The paper provides no manual verification, no random sample audit, and no comparison of flagged identifiers against a reliable second source. Without such verification, reporting raw counts as evidence of precision conflates detection with correctness. The paper even acknowledges validator limitations (OCDM-specific, SPARQL-limited for monitor; Section 6), but it does not acknowledge that the external registry dependency can produce false positives. If any large fraction of the 849/296 flags derives from registry lookup failures rather than genuinely unregistered identifiers, the 'demonstrating its precision' claim is weakened and the raw numbers lose their evidentiary value.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes two software tools developed for the OpenCitations infrastructure: oc_validator, which checks tabular metadata (META-CSV) and citation (CITS-CSV) submissions against the OpenCitations Data Model across four validation levels (wellformedness, ID syntax, ID existence, and semantics), and oc_monitor, which periodically runs SPARQL queries against OpenCitations Meta and Index to detect pre-defined data-quality issues. The tools are evaluated in two scenarios: validation of a sample of Matilda bibliographic and citation data (Tables 2 and 3) and monitoring of OpenCitations Meta with quantitative counts of affected resources (Table 4). The authors claim that the validation tool 'demonstrated its precision' and that the monitoring tool enabled detection and quantification of recurring problems in the OpenCitations Meta collection.","tokens_in":15304,"tokens_out":5091,"duration_ms":49259,"significance":"If the results are taken at face value, the paper contributes a practical, reproducible quality-assurance infrastructure for open scholarly metadata, with publicly available code, data, and a live weekly monitoring service. The evaluation against the external Matilda dataset avoids the circularity of testing on self-generated data, and the integration with GitHub Actions and a public monitoring page is a concrete operational strength. However, the central claim of 'precision' is not supported by the reported evaluation, because the paper provides no ground-truth comparison; moreover, the most frequent warning type depends on external registry lookups whose false-positive rate is unexamined. The monitoring counts are useful for internal tracking but are only as valid as the author-defined SPARQL patterns. These issues are addressable, but they currently weaken the paper's main claims.","major_comments":[{"comment":"The paper claims that the validator 'demonstrated its precision' based on counts of errors and warnings, such as 849 'br id existence' warnings in Matilda metadata and 296 in Matilda citations. In error-detection evaluation, precision is the proportion of flagged issues that are true positives, which cannot be established from raw counts alone. No manual audit, no random-sample verification, and no comparison against a reliable second source is reported for any of the flagged issues. The counts show that the tool detects patterns, but they do not demonstrate that the detections are correct. Please either add a ground-truth evaluation (for example, manual inspection of a random sample of each error type, or reconciliation against the source systems) or revise the precision claims in the abstract and Section 4.1.","section":"Abstract and §4.1, Tables 2-3"},{"comment":"The ID Existence check queries official PID registries (DOI, ORCID, PMID) to determine whether an identifier is registered. Registry outages, rate limiting, network failures, or identifiers registered after the Matilda data snapshot will cause the validator to flag valid identifiers as non-existent. The paper does not describe how such failures are handled (for example, retries, caching, or timeouts), nor does Section 6 acknowledge this external dependency, even though 'br id existence' is the most frequent warning in both Tables 2 and 3. This is load-bearing for the precision claim: if a non-negligible share of the 849 metadata warnings and 296 citation warnings are lookup failures rather than genuinely unregistered identifiers, the reported counts overstate the actual data problems. Please specify the failure-handling strategy and, where feasible, estimate the false-positive rate by re-querying or manually checking a sample.","section":"§2.2, Validation level 3 (ID Existence)"},{"comment":"The monitoring counts are produced by SPARQL queries that encode the authors' own operational definitions of issues such as 'duplicate br count' and 'br with multiple id values count'. This internal benchmarking is legitimate for tracking known issues, but the paper should state more clearly that the counts are only as valid as those patterns and should provide some verification (for example, manual review of a sample of flagged entities) before claiming that the tool enabled quantification of problems. In addition, the identical counts for 'duplicate br count' and 'duplicate id count' (1,388,761) and for 'br in multiple venues count' and 'br with multiple id values count' (760,011) suggest possible overlap between these issue definitions; the paper would benefit from explaining whether these are distinct issues that happen to coincide or the same underlying data condition counted through different queries.","section":"§4.2, Table 4"}],"minor_comments":[{"comment":"The example ORCID for Shotton, '0000-0051-5506-523X', does not match the typical ORCID identifier pattern (four groups of four digits, with the first group being a registry-assigned prefix such as 0000-0001, 0000-0002, or 0000-0003 for currently issued identifiers); please verify and correct this example, as it is used to illustrate identifier handling.","section":"§2.2, Table 1"},{"comment":"Figures 1 and 2 are introduced after Table 1, and the text says 'Table 1, Figure 2, and Figure 1'; please order the references by first mention (Table 1, Figure 1, Figure 2) or renumber the figures.","section":"§2.2, first paragraph"},{"comment":"The phrase 'more than 4,8 billion triples' uses a comma as the decimal separator; for consistency with the rest of the paper, please write '4.8 billion'.","section":"§3.2, footnote 8"},{"comment":"The final paragraph states that the evaluations 'support the reliability and applicability of the tools'; given the caveats about ground truth and external registry dependencies, this wording should be tempered or the evaluations should be strengthened accordingly.","section":"§6, Conclusions"}],"recommendation":"major_revision","confidential_remarks":"The tools are real and potentially useful, and the public code/data and live monitoring deployment are clear strengths. The main barrier to acceptance is the unsupported 'precision' claim, compounded by the unexamined false-positive risk in the ID Existence level, which affects the most frequently reported warning type. I recommend major revision rather than rejection because the gap is fillable either by adding a ground-truth audit or by substantially scaling back the claims. The identical counts in Table 4 should also be explained to rule out redundant or overlapping queries."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper reports two new open-source tools from the OpenCitations team: oc_validator, which checks incoming META-CSV and CITS-CSV tables against the OpenCitations Data Model, and oc_monitor, which runs SPARQL-based quality checks on published collections. Both tools are real, documented, and public, with evaluation material on Zenodo and a live weekly monitor page. As a software-and-evaluation report for scholarly infrastructure, it is honestly written and useful.\n\nWhat is genuinely new: the validator's four-level check structure (wellformedness, PID syntax, ID existence, semantics) with granular, human-readable error reports is a thoughtful implementation tailored to OCDM. Applying it to the Matilda dataset is a sensible external test, and the monitor's quantification of known errors in OpenCitations Meta (e.g., 1.1% of bibliographic resources sharing a persistent identifier) gives a proportionate sense of data quality. The authors also clearly state the main limitations: the validator is OCDM-specific, and the monitor only sees SPARQL-expressible issues.\n\nThe soft spot is the word \"precision\" in the abstract. The paper reports counts of flagged issues, not precision in any formal sense, and there is no ground-truth comparison. The stress-test note is right that the \"br id existence\" warnings (849 in metadata, 296 in citations) depend on external registry lookups, which can produce false positives due to registry outages, rate limits, or valid identifiers registered after the data snapshot. The authors never acknowledge this dependency as a source of error. That said, the central claim does not collapse: the tools demonstrably run and find real issues (page intervals, malformed formats, self-citations). The fix is straightforward: soften the abstract, drop \"precision,\" and ideally audit a random sample of flagged identifiers manually or against a second source. A baseline comparison with a generic RDF validator would also strengthen the evaluation, but its absence is minor.\n\nOther small points: the code repositories would benefit from explicit commit hashes for reproducibility, and the related-work section dismisses the test-driven approach [27] a little quickly, though the reasons given are reasonable.\n\nWho is this for? People working on open scholarly metadata, research assessment data, or crowdsourced ingestion pipelines. It is not a methodological breakthrough, but it is a competent engineering contribution with public artifacts. Worth serious refereeing; a careful revision should address the precision language and ideally add a small manual verification sample.","headline":"A solid, useful infrastructure report on real open-source validation and monitoring tools for OpenCitations; the abstract's \"precision\" claim overreaches because it is not backed by ground-truth evaluation.","tokens_in":15696,"tokens_out":1367,"would_cite":false,"duration_ms":16939,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenCitations' new validator and monitor aim to ensure high-quality open bibliographic and citation data by checking every incoming table against the OpenCitations Data Model and tracking known issues after publication.","keywords":["bibliographic metadata","citation data","OpenCitations","data validation","data quality monitoring","OCDM","SPARQL","persistent identifiers"],"falsifier":"Run the validator on a sample of metadata whose identifiers are known to be valid and recently registered; count how many receive a 'br id existence' warning solely because the registry has not yet indexed them. If the count is materially above zero, the precision reported for the Matilda validation is overstated.","tokens_in":14789,"feed_emoji":"🔍","tokens_out":4732,"duration_ms":41528,"temperature":0.7,"pith_summary":"The paper argues that OpenCitations, an open scholarly infrastructure, can keep its bibliographic and citation data reliable by pairing a pre-ingestion validator with a post-publication monitor. The validator checks incoming META-CSV and CITS-CSV tables against the OpenCitations Data Model using four sequential levels of rules, from table well-formedness to the existence of persistent identifiers in official registries. The monitor runs SPARQL queries against the live OpenCitations Meta and Index endpoints to detect known error patterns on a weekly schedule. Applied to a sample from the Matilda source, the validator reported 2,138 metadata issues and 358 citation issues; applied to OpenCitations Meta, the monitor quantified problems such as 1,388,761 bibliographic resources sharing a persistent identifier. If these tools work as described, OpenCitations can safely accept crowdsourced tables while keeping published data under continuous quality surveillance.","feed_headline":"Validator and monitor keep OpenCitations data clean","feed_subtitle":"Pre-ingestion checks caught 2,496 issues in Matilda; SPARQL monitor quantified recurring problems in Meta.","key_machinery":"The central object is the OpenCitations Data Model (OCDM), the set of rules defining correct relationships and properties for entities in OpenCitations Meta and Index, together with its tabular incarnation in META-CSV and CITS-CSV files. The validator's four-level pipeline is the mechanism that carries the argument: wellformedness checks table syntax; ID syntax checks PID format against issuing organizations; ID existence queries official registries such as DOI, ORCID, and PMID; semantics checks cross-field consistency like type-identifier compatibility. The monitor's machinery is a JSON-configurable set of SPARQL queries, each encoding a known error pattern, executed against the collections' endpoints and reported with pass/fail status and running time.","core_discovery":"The central claim is that a purpose-built validation tool, tailored to the OCDM's tabular format, can identify structural and semantic errors in bibliographic and citation data with enough precision to support automated rejection before ingestion, while a SPARQL-based monitoring tool can detect and quantify recurring quality problems in already published collections. The validation tool organises checks into four levels — wellformedness, ID syntax, ID existence, and semantics — and reports each failure with its exact location, validation level, error type (error vs warning), a machine-readable label, and a human-readable message. On the Matilda sample, the tool found frequent warnings for unregistered bibliographic resource identifiers (849 in metadata, 296 in citations), impossible page intervals (664), malformed page formats (522), and other issues. On OpenCitations Meta, the monitoring queries showed that 1,388,761 bibliographic resources share an identifier with at least one other resource, about 1.1% of the collection, with comparable figures for duplicate agent identifiers, demonstrating that known issues can be quantified and tracked over time.","pith_inferences":["The validator's registry-based existence checks could be tuned to treat registry failures as warnings rather than errors, avoiding false alarms when a legitimate PID is too new to be indexed.","Because the monitor only sees what is expressible in SPARQL over the endpoints, issues that appear only in dump files or API responses escape detection; a complementary dump-level audit would close that gap.","The same validation levels could be adapted to other tabular metadata schemas beyond OCDM, since wellformedness, ID syntax, existence, and semantics are generic quality dimensions.","If OpenCitations adopts crowdsourcing, the validator's output format could feed an online submission interface that highlights errors directly in the user's table, reducing friction for non-technical contributors."],"forward_implications":["Incoming tables from crowdsourced contributors can be validated automatically before ingestion, blocking malformed rows while returning precise, human-readable error reports to submitters.","Known data quality problems in OpenCitations Meta and Index can be tracked weekly, so regressions introduced by new ingestions become visible immediately.","Quantified error counts, such as the 1.1% of bibliographic resources sharing a persistent identifier, give curators a basis for prioritising manual or automated corrections.","The validator can double-check internally generated tables from the Source Preprocess step, catching conversion bugs in OpenCitations' own ingestion software before they reach published data.","The validator's non-redundancy and maximum-coverage design lets users fix all detectable issues in one pass, reducing resubmission cycles."],"supporting_citations":[{"why":"Defines the OpenCitations Data Model that the validator enforces.","marker":"[16]"},{"why":"Specifies the structure and syntax rules for META-CSV and CITS-CSV tables.","marker":"[23]"},{"why":"Provides guidance for producing well-formed CSV files for OpenCitations.","marker":"[24]"},{"why":"Describes the three-step ingestion workflow the validator precedes.","marker":"[8]"},{"why":"Describes OpenCitations Meta, the collection the monitor interrogates.","marker":"[9]"},{"why":"Supplies the Matilda dataset used as the validation case study.","marker":"[20]"},{"why":"Provides the input data and results of the two applicative scenarios.","marker":"[25]"}],"fun_headline_variants":["OpenCitations validator flags 2,496 issues in test source","2,496 data issues found by OpenCitations validation tool","OpenCitations: validator and monitor for data quality","OpenCitations' new tools catch and track data errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The validator's third level assumes that official persistent identifier registries such as DOI, ORCID, and PMID are complete, current, and always reachable at validation time, so a registry query failure or a not-yet-registered identifier would be reported as a false error.","fun_headline_variants_meta":{"raw":{"variants":["OpenCitations validator flags 2,496 issues in test source","2,496 data issues found by OpenCitations validation tool","OpenCitations: validator and monitor for data quality","OpenCitations' new tools catch and track data errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001102,"raw_usage":{"total_tokens":4631,"prompt_tokens":1015,"completion_tokens":3616,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":3551}},"tokens_in":631,"tokens_out":3616,"duration_ms":30111,"temperature":1.0,"reasoning_tokens":3551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:34:46.534185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the validator on a sample of metadata whose identifiers are known to be valid and recently registered; count how many receive a 'br id existence' warning solely because the registry has not yet indexed them. If the count is materially above zero, the precision reported for the Matilda validation is overstated.","supporting_citations":[{"cited_title":"In: Pan, J.Z., Tamma, V., d’Amato, C., Janowicz, K., Fu, B., Polleres, A., Senevi- ratne, O., Kagal, L","cited_arxiv_id":null,"evidence_quote":"Defines the OpenCitations Data Model that the validator enforces."},{"cited_title":"How to structure citations data and bibliographic metadata in the OpenCitations accepted format","cited_arxiv_id":"2206.03971","evidence_quote":"Specifies the structure and syntax rules for META-CSV and CITS-CSV tables."},{"cited_title":"Zenodo (2022)","cited_arxiv_id":null,"evidence_quote":"Provides guidance for producing well-formed CSV files for OpenCitations."},{"cited_title":"Scientometrics (2024) https: //doi.org/10.1007/s11192-024-05160-7 21","cited_arxiv_id":null,"evidence_quote":"Describes the three-step ingestion workflow the validator precedes."},{"cited_title":"In: ELPUB 2019 23d International Confer- ence on Electronic Publishing","cited_arxiv_id":null,"evidence_quote":"Supplies the Matilda dataset used as the validation case study."},{"cited_title":"Zenodo (2025)","cited_arxiv_id":null,"evidence_quote":"Provides the input data and results of the two applicative scenarios."}],"review_version":1}