{"id":"4e9a524e-9cb3-4dba-bbb5-2047d1d7f1f1","arxiv_id":"2504.19023","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"GLaMoR uses a Graph Language Model on triple-converted OWL modules to identify logically inconsistent ontologies with about 95% accuracy, beating several text and embedding baselines.","lead":"GLaMoR converts OWL ontologies into graph-plus-text triples and trains a Graph Language Model to flag logically inconsistent modules, reaching about 95% accuracy on small modules. It is a useful test of whether language models can serve as fast approximate stand-ins for exact ontology reasoners.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significance test supports the headline 'outperforms all baselines': gGLM-base vs ModernBERT accuracy differs by only 0.86 pp with overlapping standard deviations across 5 runs.","rationale":"The paper is a competent empirical study and its main quantitative claims are mostly internally consistent: the test-set labels are verified by HermiT (Table IV, 100/100/100), the runtime advantage is real even under the paper's own conservative accounting, and the robustness study is a genuine generalization probe. However, the abstract and Discussion claim that the global GLM 'outperforms all baseline models.' The gap over ModernBERT on the primary metric is 0.86 pp with overlapping dispersion over 5 runs, and no significance testing is reported. Since a central contribution is comparative, this is the weakest point. It is fixable by reporting paired bootstrap CIs or running more seeds, and it does not undermine the high absolute accuracy or the strong speed advantage. Hence the verdict remains CONDITIONAL.","tokens_in":16400,"tokens_out":6867,"duration_ms":74216,"concrete_test":"Recover the per-fold predictions of gGLM-base and ModernBERT for each of the 5 runs, or rerun both models with 30 seeds on identical train/validation/test splits, and compute a paired bootstrap 95% confidence interval for the accuracy difference. If the interval excludes 0, the 'outperforms all baselines' claim survives; if it includes 0, the claim must be weakened to 'comparable to ModernBERT, better than all other baselines.' Report the same test for precision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the global GLM 'outperforms all baseline models' (Abstract; Section VI-A) rests on a 0.86 percentage-point accuracy gap over ModernBERT (95.13 ± 1.10 vs 94.27 ± 2.09, n = 5 runs; Table IV). With these standard deviations, the gap is well within sampling noise; the paper reports no paired test, confidence interval, or significance value. The same holds for precision (96.10 ± 1.97 vs 95.82 ± 2.31). Because the central contribution is explicitly comparative, this unsupported gap is the most load-bearing weakness. The speed claim is on firmer ground: even the total ML runtime (training + inference, about 4.8 h for gGLM-base) is roughly 25× less than HermiT's 122 h, and inference alone is far faster. The reader's label-verification worry is largely mitigated by Table IV itself: HermiT is reported at 100/100/100 on the test set, meaning its logical results exactly match the ground-truth labels on the test set; unverified labels in the training split would add noise rather than inflate test metrics. The remaining weakness is statistical.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GLaMoR, a pipeline that turns OWL ontologies from the NCBO BioPortal repository into modules using OAPT, generates inconsistent examples by injecting axioms from 14 anti-patterns, translates Manchester syntax into English triples, and trains T5-based Graph Language Models (GLMs) in local and global settings to classify module consistency. The authors compare against classical machine-learning baselines on OWL2Vec* embeddings, PRODIGY, WideMLP, ModernBERT, LongT5, Llama3 in a zero-shot setting, and the HermiT reasoner, reporting that the global GLM with T5-base achieves 95.13% accuracy while HermiT takes 122 hours on the test set. A robustness study withholds inconsistency families from training to gauge generalization, and the code is released.","tokens_in":16587,"tokens_out":5312,"duration_ms":50307,"significance":"If the reported results are statistically sound, the work is a useful step toward scalable approximate consistency checking and contributes a new dataset of ontology modules with injected inconsistencies. The manuscript is transparent in releasing code, reports five-run averages with standard deviations, and evaluates on a held-out test set, and the robustness study is a good generalization check. The main caveats are that the headline comparison with ModernBERT is not supported by significance testing, that the hyperparameter protocol appears to use the test split for model selection, and that the token-length filter discards a large share of the data without analysis of representativeness.","major_comments":[{"comment":"The hyperparameter optimization protocol states that for several models the authors \"used a (70, 15, 15) split, and chose the parameters for which the accuracy on the test split was the highest.\" If the final numbers in Table IV are measured on that same test split, the reported accuracy is optimistically biased and the comparison between models is invalid. Please clarify whether a separate validation split was actually used; if not, model selection and final evaluation must be separated (e.g., by selecting on the validation split or using nested cross-validation) and the tables recomputed.","section":"Section IV-C"},{"comment":"No statistical test accompanies the claim that the global GLM \"outperforms all baseline models\" (Abstract and Section VI-A). The accuracy gap between gGLM-base (95.13 ± 1.10) and ModernBERT (94.27 ± 2.09) is 0.86 percentage points, which is within one standard deviation of both run sets; the precision gap (96.10 ± 1.97 vs 95.82 ± 2.31) is similarly small. Please add a paired significance test (e.g., Wilcoxon signed-rank over the five runs) or confidence intervals, and temper the comparative claims in the Abstract and Section VI-A accordingly.","section":"Section V, Table IV"},{"comment":"The 4,096-token filter discards 3,336 of 7,505 consistent modules (44%) and 9,823 of 19,902 inconsistent modules (49%). The paper does not analyze whether accuracy and runtime generalize to the discarded larger modules; as it stands, the speed comparison with HermiT and the accuracy claims apply only to modules of at most 4,096 tokens. Please report performance as a function of module size or provide evidence that the filtered subset is representative of the full modularized corpus.","section":"Section IV-A, Table III"},{"comment":"The 7,505 \"consistent\" modules produced by OAPT are not explicitly verified with a reasoner. Although HermiT's 100% accuracy on the test set in Table IV implies that the test labels are correct, the released dataset's training labels should be verified (or a statement added about whether and how verification was performed), because label noise in the training split would propagate to the reported robustness analysis and to the usability of the released dataset.","section":"Section IV-A"}],"minor_comments":[{"comment":"The claim \"20 times faster\" does not match the reported numbers: HermiT takes 122:24:32.42, while the slowest ML runtime in Table IV is about 4.75 hours for gGLM-base, giving a ratio of roughly 25.7. Please state the exact ratio or use the \"under six hours\" formulation consistently.","section":"Abstract, Section VI-A, Table IV"},{"comment":"The caption should state explicitly whether the test set still contains the withheld inconsistency family. The text implies that it does, but this is a critical detail for interpreting the robustness experiment.","section":"Table V"},{"comment":"The sentence \"with only 3% of classifications of inconsistent ontologies that are actually consistent ontologies\" is confusing; please rephrase in terms of the false-positive rate derived from precision (96.10% implies approximately 3.9% false positives).","section":"Section VI-A"},{"comment":"Typos and formatting issues: \"Graph Lanugage Model\" in Section I, \"Modularizaition\" in Appendix A, and \"i.¸ e.\" in Appendix D.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The test-split selection issue in Section IV-C is potentially disqualifying if it is not a wording error; please ask the authors to clarify the exact protocol and, if necessary, rerun the experiments with a proper validation procedure. The significance-testing gap is also important for the journal's stated claims. The dataset and code release are valuable, so I prefer major revision over rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a decent empirical paper with a real dataset contribution, but the headline 'outperforms all baselines' isn't backed by any significance testing. The 0.86 pp accuracy gap over ModernBERT (95.13 ± 1.10 vs 94.27 ± 2.09) is within sampling noise. The robustness study is the strongest part.\n\nWhat's new: first application of Graph Language Models to OWL consistency checking; a new dataset of 19,902 synthetic inconsistent modules generated from 14 anti-patterns; and a robustness analysis that withholds entire inconsistency families. The code is public. Global GLM with T5-base reaches ~95% accuracy, which is far better than embedding-based classifiers (all below 62%), and the global setting clearly beats the local one. The robustness result—global GLMs stay above 90% accuracy when whole families like UE* or CSC are absent from training—is a genuine generalization test, not just a held-out split. That deserves credit.\n\nSoft spots, in order of severity. First, the comparative claim. The paper says the global GLM 'outperforms all baseline models' in the abstract and discussion, but the difference from ModernBERT is not significant given n=5 runs and overlapping standard deviations. No paired test, no CI. This is a fixable but load-bearing flaw.\n\nSecond, the runtime comparison. They compare total ML runtime (training + inference) against HermiT's inference-only time. HermiT doesn't train, so the '20x faster' framing understates the cost of building the model. They are explicit about what they measure, but a reader could easily over-read it.\n\nThird, the 4,096-token filter: they discard nearly half of the consistent and inconsistent modules. So the result only speaks to small ontology modules. They acknowledge this in future work.\n\nFourth, the consistent labels from OAPT are not verified with a reasoner. The stress-test note points out, correctly, that HermiT's perfect 100/100/100 on the test set corroborates the test labels, so this is mostly a potential noise source in training, not a fatal flaw. Still, worth asking for a verification step.\n\nFifth, the anti-patterns are synthetic; real-world inconsistencies may differ. They list this as a threat to validity.\n\nBottom line: the paper is worth engaging with. The dataset and robustness study are valuable, and the empirical gap to ModernBERT, while not significant, is plausible. A serious referee should ask for significance testing, a runtime breakdown, and ideally a verification of the consistent-module labels. I'd send it to review rather than desk reject.","headline":"A useful empirical study with a solid dataset and robustness analysis, but the headline accuracy claim over ModernBERT is within noise and needs statistical support.","tokens_in":17151,"tokens_out":2930,"would_cite":false,"duration_ms":27356,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An adapted Graph Language Model can classify OWL ontologies as consistent or inconsistent with 95% accuracy, and it does so far faster than a classical reasoner.","keywords":["consistency checking","OWL ontologies","Graph Language Model","semantic reasoning","graph classification","anti-pattern injection","ontology modularization"],"falsifier":"Run a standard OWL 2 reasoner over the 7,505 modules labeled consistent and see whether any are judged inconsistent; if even one is, the paper's ground truth is contaminated. As a second check, test the trained global GLM on naturally occurring inconsistent ontologies that were not built from the 14 injected anti-patterns and compare its labels with reasoner labels.","tokens_in":16145,"feed_emoji":"🧠","tokens_out":8585,"duration_ms":87848,"temperature":0.7,"pith_summary":"This paper proposes GLaMoR, a pipeline that treats OWL ontology consistency checking as a binary graph classification problem instead of a purely logical one. Large ontologies are split into smaller modules, a balanced set of inconsistent examples is created by injecting logical anti-patterns, and each module is verbalized as English-like triples for a Graph Language Model. On a dataset built from a large biomedical ontology repository, the best model—a global-attention GLM with T5-base—reaches 95.13% accuracy and the highest precision among all compared models, beating classical machine-learning baselines and several modern language and graph models. The same model completes training and inference in under six hours, while the classical reasoner used for comparison spends 122 hours on the test set. If the result holds, learned models become a practical fast screening tool for ontology consistency, complementing or replacing reasoners where speed matters.","feed_headline":"Graph language model checks ontologies at 95% accuracy","feed_subtitle":"It beats every tested baseline and finishes in under six hours what a classical reasoner spends 122 hours on.","key_machinery":"The central object is the Graph Language Model: a Transformer whose attention is augmented with graph-structural positional encodings and initialized from T5 weights, allowing an ontology to be read as a graph rather than a flat string. Each OWL module is first converted into a Levi graph—one node per entity and per relation, with relation nodes connecting their head and tail—then into token sequences. The global-attention setting lets any node attend to any other node, while the local setting restricts attention to a single triple. Mean pooling over the model's output logits turns the sequence into a binary consistency score. The anti-pattern injection rules and the modularization step supply the training signal and keep inputs within the model's 4,096-token limit, but the GLM's graph-aware attention is the mechanism the paper credits for detecting contradictions spread across multiple axioms.","core_discovery":"The paper claims that a Graph Language Model initialized from T5 and augmented with graph positional encodings can classify OWL ontology modules as consistent or inconsistent when the ontology is serialized as a sequence of triples. On a balanced dataset built from modularized biomedical ontologies and 14 injected logical anti-patterns, the global GLM with T5-base reaches 95.13% accuracy, 96.10% precision, and 94.17% recall, the best accuracy and precision among all machine-learning baselines and better than local-attention GLMs. The paper also reports that the same model keeps accuracy above 90% when whole families of anti-patterns are withheld from training, while local models degrade sharply on cyclic patterns; the authors attribute this to global attention's ability to connect axioms across the entire graph. This is presented as evidence that consistency checking is not the exclusive domain of logical reasoners.","pith_inferences":["A fair deployment comparison would amortize the model's one-time training cost over many ontologies; the paper's runtime advantage is a bulk-processing advantage, not necessarily a single-ontology one.","Because the verbalized triples are English sentences, part of the GLM's accuracy may come from T5's language priors rather than from graph structure; ablating the translation step to compare raw triples against English would separate the two contributions.","The strong global-over-local gap suggests the model's attention patterns could be inspected to see whether it learns to trace multi-axiom contradiction chains, connecting this work to explainable reasoning in description logics."],"forward_implications":["Consistency checking can be framed as binary graph classification, so the same triple-based pipeline could be retrained for other ontology properties without a reasoner.","Global attention is doing the heavy lifting: local GLMs lag behind and drop sharply on cyclic inconsistencies, so any learned reasoner for ontologies should attend over the whole graph rather than per-axiom.","The best model's wall-clock advantage—under six hours of training plus minutes of inference versus 122 hours for the classical reasoner—makes learned consistency screening practical for bulk ontology repositories.","Withholding an inconsistency family from training still leaves the global GLM above 90% accuracy, suggesting the model learns general contradiction-detecting structure rather than memorizing the 14 injected patterns."],"supporting_citations":[{"why":"Supplies the Graph Language Model architecture that GLaMoR adapts for sequence classification.","marker":"[11]"},{"why":"Defines the logical anti-patterns used to inject inconsistencies into consistent ontology modules.","marker":"[12]"},{"why":"Provides the OAPT modularization tool that splits large ontologies into the modules used as data points.","marker":"[29]"},{"why":"Supplies the ontology corpus that forms the basis of the consistency-checking dataset.","marker":"[41]"},{"why":"Provides OWL2Vec* embeddings used to train all embedding-based baseline models.","marker":"[35]"},{"why":"Supplies the classical reasoner baseline whose accuracy and runtime are the comparison targets.","marker":"[3]"},{"why":"Supports the choice of the classical reasoner as a strong and efficient baseline.","marker":"[4]"},{"why":"Provides the T5 model whose weights initialize the Graph Language Model.","marker":"[38]"},{"why":"Supplies the strongest text-only baseline that the global GLM must outperform.","marker":"[19]"}],"fun_headline_variants":["Ontology consistency via graph language model hits 95%","Graph LLM checks OWL ontologies at 95% accuracy","GLaMoR: 95% accuracy on ontology consistency, 20x faster","T5-based GLM achieves 95% on ontology consistency tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that all 7,505 modules produced by the modularization tool are truly consistent; the paper labels them consistent without running a reasoner over them, so if any contains a contradiction the ground-truth labels and all reported metrics are corrupted.","fun_headline_variants_meta":{"raw":{"variants":["Ontology consistency via graph language model hits 95%","Graph LLM checks OWL ontologies at 95% accuracy","GLaMoR: 95% accuracy on ontology consistency, 20x faster","T5-based GLM achieves 95% on ontology consistency tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2581,"prompt_tokens":939,"completion_tokens":1642,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":1565}},"tokens_in":555,"tokens_out":1642,"duration_ms":10125,"temperature":1.0,"reasoning_tokens":1565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:03:00.300134+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a standard OWL 2 reasoner over the 7,505 modules labeled consistent and see whether any are judged inconsistent; if even one is, the paper's ground truth is contaminated. As a second check, test the trained global GLM on naturally occurring inconsistent ontologies that were not built from the 14 injected anti-patterns and compare its labels with reasoner labels.","supporting_citations":[{"cited_title":"Bioportal: enhanced functionality via new web services from the national center for biomedical ontology to access and use ontologies in software applications,","cited_arxiv_id":null,"evidence_quote":"Supplies the ontology corpus that forms the basis of the consistency-checking dataset."},{"cited_title":"Owl2vec*: Embedding of owl ontologies,","cited_arxiv_id":null,"evidence_quote":"Provides OWL2Vec* embeddings used to train all embedding-based baseline models."},{"cited_title":"Hermit: An OWL 2 reasoner,","cited_arxiv_id":null,"evidence_quote":"Supplies the classical reasoner baseline whose accuracy and runtime are the comparison targets."},{"cited_title":"OAPT: A tool for ontology analysis and partitioning,","cited_arxiv_id":null,"evidence_quote":"Provides the OAPT modularization tool that splits large ontologies into the modules used as data points."},{"cited_title":"Graph language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the Graph Language Model architecture that GLaMoR adapts for sequence classification."},{"cited_title":"Catalogue of anti- patterns for formal ontology debugging,","cited_arxiv_id":null,"evidence_quote":"Defines the logical anti-patterns used to inject inconsistencies into consistent ontology modules."},{"cited_title":"Comparison of reasoners for large ontologies in the OWL 2 EL profile,","cited_arxiv_id":null,"evidence_quote":"Supports the choice of the classical reasoner as a strong and efficient baseline."}],"review_version":1}