{"id":"74772f6b-6110-4a11-a9ba-f5fc83413ee2","arxiv_id":"2411.15550","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A census of class-order violations in Wikidata found millions of conflicting statements, and a small set of manual fixes meaningfully reduced the number of high-order class anomalies.","lead":"Wikidata, the open knowledge base, contains millions of class-hierarchy statements that violate its own intended class ordering, such as classes that are both instances and subclasses of the same class. The authors use fast SPARQL queries to count these problems, manually fix a few hundred of them, and show that the fixes reduce some of the worst anomalies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Split-order enumeration rests on an unverified query decomposition: QLever cannot run the full possibly-empty path query, so the 'completely enumerate' claim depends on an equivalence that is asserted, not proven.","rationale":"The reader's stated weakest assumption is the intended semantics of class order in Wikidata. That is a real concern, but the paper mostly uses its own explicit definitions and labels many outputs as 'suspect' rather than definitively wrong, so the semantic premise is at least internally coherent. The more directly load-bearing issue for the paper's strongest claim is whether the split-order enumeration is actually complete, since 'completely enumerate' is an objective computational claim and the manuscript itself acknowledges that QLever cannot run the full query. The reader did mention split-order completeness in the rationale, but not as the primary weakest assumption, hence 'partial' agreement. The paper has independent support: the queries are provided, the manual fixes are documented with dates and batches, and the authors are transparent about the confounding effect of concurrent edits. Those strengths do not remove the need for an equivalence check on the decomposition. Since the reader's CONDITIONAL verdict already hinges on completeness and reproducibility concerns, this stress-test does not move the verdict; it sharpens the condition under which the result should be accepted.","tokens_in":18912,"tokens_out":5229,"duration_ms":51620,"concrete_test":"Recompute the split-order pairs from the same June 2024 Wikidata dump without relying on QLever's missing zero-length path support. Load the wdt:P31 and wdt:P279 edges into a graph library, compute the reflexive-transitive closure of P279, and evaluate the full predicate: ?c P279* ?s AND ?c P279* ?x AND ?x P31 ?y AND ?y P279* ?s. Compare the resulting (c,s) pair set to the union-minus-exclusions output of the paper's pipeline, and compare the 6,379 distinct s values. Independently verify the exclusion criterion by checking that every removed pair has either a superclass of c or a subclass of s that certifies the same split; if any removed pair lacks such a certificate, the headline split-order count is wrong.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that QLever can completely enumerate certain class-order problems, and a key numerical output is the 6,379 split-order classes. That enumeration is load-bearing and its weakest point is the split-order query decomposition in 'Classes That Cannot Have Fixed Order' and the Technical Appendix. The intended full query contains a possibly-empty path, ?c wdt:P279*/wdt:P31/wdt:P279* ?s, which QLever cannot evaluate. The authors split it into self/AB/C/D cases, run exclusion subqueries, and combine results with comm, reporting 84,158,269 raw pairs and, after exclusions, 3,821,300 pairs and 6,379 distinct split-order classes. No formal containment or equivalence proof is given that the union of the subqueries minus the exclusions equals the original query. The exclusions remove roughly 96% of the raw pairs, so a small error in a subcase or in the comm-based subtraction would change the headline 6,379 count and the table of classes with the most split-order items. This is not a dispute about Wikidata semantics: even granting the authors' intended meaning of P31 and P279, the completeness of the enumeration itself is unverified. The paper should be credited for publishing the queries and for being transparent that archived dumps are available only by contacting the first author, but 'complete enumeration' is exactly the kind of claim that requires either a machine-checkable equivalence proof or an independent reproduction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies class-order violations in Wikidata as of June 2024. It defines fixed class order using the instance-of (P31) and subclass-of (P279) properties, counts classes whose order is directly determined by Wikidata's built-in fixed-order classes, computes minimum possible orders from instance chains, and enumerates split-order classes and instance loops. The authors report 3,238 third-order, 2,447,483 second-order, and 14,062,244 first-order classes with millions of order overlaps, 6,379 split-order classes, and 120 instance loops. They made a small number of manual edits to Wikidata, reran the main queries, and observed reductions in high-minimum-order classes. They conclude that QLever can completely enumerate certain class-order problems and offer suggestions for tooling and community processes.","tokens_in":19148,"tokens_out":17805,"duration_ms":153343,"significance":"If the counts and the completeness of the queries are correct, the paper provides the most extensive quantification to date of class-order problems in Wikidata, updating and generalizing earlier anti-pattern work. The authors are transparent: the queries are in the appendix, the use of QLever is a practical contribution, and the manual fixes demonstrate that some problems are tractable. The instance-chain queries provide evidence independent of Wikidata's own fixed-order classes. The main weaknesses are that the split-order enumeration lacks a proof of equivalence to the intended full query, and the instance-loop query is acknowledged to have missed loops with two instance links. These weaknesses bear directly on the 'complete enumeration' claim, so they need to be addressed before the numerical results can be fully relied upon.","major_comments":[{"comment":"The paper's summary claims that QLever can 'completely enumerate' class-order problems, and the split-order count of 6,379 distinct classes is a headline result. The full split-order query in the Technical Appendix contains the possibly-empty path wdt:P279*/wdt:P31/wdt:P279*, which the authors state QLever cannot evaluate. They therefore split the query into self/AB/C/D subcases, run exclusion queries, and combine the results with comm. However, no proof is given that the union of the four subcases, after subtracting the exclusion sets, is exactly the answer of the original query. Because the exclusions remove about 80 million of the 84,158,269 raw pairs, leaving 3,821,300 pairs, an error in any subcase or in the comm-based subtraction would change the reported 6,379 classes and the ranked table in the 'Split-Order Classes' analysis. I would like to see either a formal containment/equivalence argument for the decomposition, a verification on a data graph small enough to be checked exhaustively, or an independent run on an engine that supports arbitrary-length paths.","section":"Classes That Cannot Have Fixed Order (and Technical Appendix)"},{"comment":"The instance-loop query presented in the Technical Appendix only detects cycles of the form ?c wdt:P31 ?c and ?c wdt:P31/wdt:P279+ ?s with ?c = ?s. The paper later reports that 'there where instance loops that were not detected in the initial queries' and identifies loops with two instance links, such as legal abuse (Q6517445) being an instance of abuse of rights (Q12586068) and vice versa. The initial count of 120 items in instance loops is therefore not a complete enumeration of the kind of problem the paper says it is enumerating, and the before/after minimum-order counts after fixing loops are presented as reductions even though the query did not capture all loops. This needs to be reconciled with the summary's completeness claim: either supply queries covering pure P31/P31 cycles, or weaken the claim for this category.","section":"Instance Loops (and Technical Appendix)"}],"minor_comments":[{"comment":"Because other editors may have changed Wikidata during the study, the before/after comparisons are not controlled. The authors acknowledge this, but the section should state explicitly that the reported reductions are not solely attributable to the described fixes.","section":"Resulting Counts"},{"comment":"Reproducibility would be improved by depositing the archived dumps and result files with a persistent identifier rather than making them available 'by contacting the first author'.","section":"Technical Appendix"},{"comment":"There is a typo 'human (QA5)' that should be 'human (Q5)'.","section":"Split-Order Classes"},{"comment":"There are several typos and ungrammatical passages, including 'caes', 'sucessfully', 'minimim-order', and the sentence beginning 'It is useful to exclude items that have where the item has a superclass with...'.","section":"Technical Appendix"}],"recommendation":"major_revision","confidential_remarks":"The split-order decomposition gap and the acknowledged incompleteness of the instance-loop query are the main technical barriers to accepting the paper's completeness claims. Both issues are fixable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a genuine empirical contribution—current (2024) counts of class-order violations in Wikidata, generalized anti-patterns, and a real attempt to fix some of them—but the paper's strongest claim, that QLever can completely enumerate split-order classes, is not fully supported by what's in the appendix. The central finding that Wikidata has millions of class-order problems is robust; the exact split-order number is less certain.\n\nWhat's new: it updates Brasileiro/Dadalto's older counts with contemporary data, introduces instance loops and minimum-order chains, and reports before/after effects of manual fixes. The queries are published in the appendix, which is a real credit. The authors are transparent about the confounding effect of concurrent edits. The 120 instance-loops analysis and the fix effects are credible. The paper is honest about what is evidence and what is just 'suspect'.\n\nSoft spots, in order of weight:\n\n1. The split-order enumeration. The intended query has a possibly-empty path that QLever can't run. The authors split it into cases and subtract exclusions using `comm`, but give no proof—formal or otherwise—that the union of subcases minus exclusions equals the intended query. The raw result is 84M pairs, and after exclusions only 3.8M pairs and 6,379 classes remain. An error in any subcase or in the subtraction would change that headline number. This is a completeness claim, so the burden is on the authors to verify the decomposition. Independent reproduction or a machine-checkable equivalence would settle it.\n\n2. Data availability. Archived dumps and query results are available only by contacting the first author. For a paper whose claims are counts, that's too weak. A public archive would make the numbers checkable.\n\n3. The semantic premise. The paper assumes the intended meaning of P31/P279 includes a strict class-order hierarchy. That's a reasonable reading of Wikidata norms, and the authors acknowledge Wikidata has no formal semantics, but it means 'violations' here are violations of a community convention, not of logic. This is a caveat, not a fatal flaw.\n\nThis is a solid empirical paper for the Wikidata and KG-quality community. It deserves a serious referee, but I'd want the split-order decomposition either proven or reproduced before trusting 6,379. If that's fixed, it's worth citing and useful for anyone working on ontology cleaning in Wikidata.","headline":"Useful, current empirical survey of Wikidata class-order problems, but the 'complete enumeration' of split-order classes rests on an unverified query decomposition.","tokens_in":19707,"tokens_out":1993,"would_cite":true,"duration_ms":18162,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A complete census of Wikidata's class-order violations finds millions of conflicts.","keywords":["class order","Wikidata","SPARQL","ontology quality","instance of","subclass of","multi-level modeling","knowledge graph"],"falsifier":"Sample a few hundred of the reported split-order and multi-order classes and ask domain editors whether the conflicting statements are intentional. If a large fraction are defended as correct modeling (for example, a biological 'gene' class used both as a class of genes and as an instance of a gene-concept class), the paper's claim that these are problems to be fixed is weakened. Alternatively, a controlled experiment could take two identical Wikidata dumps, apply the paper's proposed editor warnings to one, and measure whether new class-order conflicts accumulate more slowly.","tokens_in":18696,"feed_emoji":"🗃️","tokens_out":4840,"duration_ms":38704,"temperature":0.7,"pith_summary":"This paper establishes that the class-order problems long suspected in Wikidata can be fully enumerated with current query tools, and that the resulting counts are enormous: as of June 2024, 3,238 third-order classes, 2,447,483 second-order classes, and 14,062,244 first-order classes, with millions of overlapping memberships that violate the rule that a non-empty class has at most one fixed order. It also counts 120 instance loops and 6,379 split-order classes that cannot have a fixed order at all. The authors manually fixed a small number of these problems in Wikidata itself and showed that a handful of edits can cut some problem counts substantially, for example reducing the number of classes with minimum order four or more by roughly half. The wider point is that the Wikidata ontology's intended semantics for 'instance of' and 'subclass of' are not being enforced, and that better tooling and community processes could address this.","feed_headline":"Millions of Wikidata classes break their own order rules","feed_subtitle":"A SPARQL census finds 3,238 third-order and 14 million first-order classes, with millions of conflicts; a few edits cut some counts in half.","key_machinery":"The central object is class order, a hierarchy on Wikidata classes: non-classes have order zero, a first-order class has no classes as instances, a second-order class has only first-order classes as instances, and an nth-order class has only (n−1)th-order classes as instances. The paper exploits five built-in Wikidata classes—first-order class, second-order class, third-order class, fourth-order class, and fifth-order class—whose intended meaning fixes the order of their instances and subclasses, and uses SPARQL queries over the RDF dump to enumerate classes that are required to have a fixed order, classes whose order is bounded by instance chains, and split-order classes. The queries, run on a fast SPARQL engine, are the machinery that makes complete enumeration feasible.","core_discovery":"The central discovery is that class order in Wikidata, defined by the rule that a first-order class has no classes as instances, a second-order class has only first-order classes as instances, and so on, can be checked exhaustively. Using SPARQL queries against the Wikidata RDF dump, the paper reports concrete counts: 3,238 third-order classes, 2,447,483 second-order classes, and 14,062,244 first-order classes, with 3,159 third-order classes also claimed to be second-order and 2,386,595 second-order classes also claimed to be first-order. It also identifies 120 instance loops and 6,379 split-order classes (classes having a subclass that is also a subclass of an instance of the class). A small set of manual fixes, such as removing a second-order parent from a large chemistry class and breaking 113 incorrect instance loops, reduced the count of classes with minimum order four from 2,854 to 1,924 and the count at order six from 1,413 to 722.","pith_inferences":["If the class-order semantics are taken as the intended meaning, the count of 14 million first-order classes suggests that many items currently modeled as classes in Wikidata (e.g., lists and sequences) may not be genuine classes; the paper hints at this but does not push it.","The same enumeration technique could be applied to other collaborative knowledge bases or to later Wikidata dumps, turning the reported numbers into a periodic quality metric.","The large overlap between second- and first-order classes implies that many Wikidata queries that rely on the class hierarchy, such as retrieving all instances of a class, can silently return both individuals and classes, which would matter for downstream applications."],"forward_implications":["The reported counts give a baseline: any future edit or tooling change to Wikidata can be measured against the June 2024 numbers.","Because a few edits (removing one parent class, breaking 113 loops) cut some problem counts roughly in half, further targeted fixes could plausibly reduce the largest conflicts without a full rewrite of the ontology.","The 3,037 split-order classes caused by a single item each are candidates for manual or semi-automatic repair, while classes like gene and protein, each with hundreds of thousands of split-order entries, will require community-level modeling decisions.","If Wikidata adopted the proposed safeguards—warning editors when an edit creates an instance loop or class-order conflict, and upgrading the query service—the rate at which new order violations accumulate could drop."],"supporting_citations":[{"why":"Introduces Wikidata as the collaborative knowledge base under study, framing its structure and intended use.","marker":"Vrandečić and Krötzsch 2014"},{"why":"Provides the fast SPARQL engine that makes exhaustive enumeration of the large result sets feasible.","marker":"Bast and Buchhold 2017"},{"why":"Supplies the multi-level modeling theory and earlier counts of class-order anti-patterns that this work updates and generalizes.","marker":"Brasileiro et al. 2016"},{"why":"Re-measures the anti-patterns five years later and finds dramatic increases, giving the paper a baseline for comparison.","marker":"Dadalto et al. 2021"},{"why":"Documents the known ontology issues in Wikidata, motivating the need for systematic fixes.","marker":"Abdulai and Lacroix 2023"},{"why":"Provides evidence that instance-of and subclass-of links are often misused, supporting the interpretation of the counted violations as errors.","marker":"Shenoy et al. 2022"}],"fun_headline_variants":["Wikidata class disorder: millions of classes in wrong order","SPARQL census finds millions of Wikidata class-order conflicts","Manual fixes halve Wikidata class order violations","Millions of Wikidata classes break their own hierarchy rules"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that Wikidata editors intend 'instance of' and 'subclass of' to form a strict class-order hierarchy, so that a class cannot be both a first-order and a second-order class; if many editors legitimately use classes at multiple levels, the counted 'violations' are not necessarily errors.","fun_headline_variants_meta":{"raw":{"variants":["Wikidata class disorder: millions of classes in wrong order","SPARQL census finds millions of Wikidata class-order conflicts","Manual fixes halve Wikidata class order violations","Millions of Wikidata classes break their own hierarchy rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1708,"prompt_tokens":845,"completion_tokens":863,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":800}},"tokens_in":461,"tokens_out":863,"duration_ms":8247,"temperature":1.0,"reasoning_tokens":800,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:09:51.770256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample a few hundred of the reported split-order and multi-order classes and ask domain editors whether the conflicting statements are intentional. If a large fraction are defended as correct modeling (for example, a biological 'gene' class used both as a class of genes and as an instance of a gene-concept class), the paper's claim that these are problems to be fixed is weakened. Alternatively, a controlled experiment could take two identical Wikidata dumps, apply the paper's proposed editor warnings to one, and measure whether new class-order conflicts accumulate more slowly.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the fast SPARQL engine that makes exhaustive enumeration of the large result sets feasible."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multi-level modeling theory and earlier counts of class-order anti-patterns that this work updates and generalizes."},{"cited_title":"A.; Almeida, J","cited_arxiv_id":null,"evidence_quote":"Re-measures the anti-patterns five years later and finds dramatic increases, giving the paper a baseline for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that instance-of and subclass-of links are often misused, supporting the interpretation of the counted violations as errors."}],"review_version":1}