Pith. sign in

REVIEW 3 major objections 5 minor 12 references

A two-stage LLM workflow turns French maintenance regulations into ontology-grounded knowledge graphs with near-perfect class alignment and far fewer duplicate entities after fusion.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 11:49 UTC pith:5OVOYDRH

load-bearing objection Clean end-to-end engineering poster on French maintenance regs: strong structural hygiene, real artifacts, but fusion and correctness are still open. the 3 major comments →

arxiv 2607.24551 v1 pith:5OVOYDRH submitted 2026-07-27 cs.AI

LLM-Assisted Ontology Engineering and Construction of a French Legal Knowledge Graph

G{\'e}nesis Montenegro (WIMMICS) , Mokhtar Boumedyen Billami , Catherine Faron (WIMMICS) , Fabien Gandon (WIMMICS) , Pierre Monnin (WIMMICS) This is my paper
classification cs.AI
keywords knowledge graph constructionontology engineeringlanguage modelslegal textsmaintenance regulationembedding-based fusionrelation signature
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

French industrial maintenance rules are dense legal texts that are hard to query for a specific case or to plug into operational systems. This paper shows a practical two-stage method that starts from a small core legal ontology, lets language models openly extract typed entities and relations from a stratified sample, normalizes labels by embedding similarity, and induces candidate object properties with their domain–range signatures. The resulting ontology then guides closed triple extraction over the full corpus of more than six thousand articles, which is lifted into RDF. Experiments with two large models produce fully valid structured outputs, almost complete class alignment, and a sharp drop in duplicated entities and predicates after fusion; fewer than one in five triples introduce previously unseen properties. The remaining gaps are mainly new domain–range combinations for already-known predicates, so the authors identify predicate normalization and signature validation as the next industrial refinement steps.

Core claim

An LLM-assisted two-stage pipeline—open extraction plus embedding fusion and signature-driven property induction on a sample, followed by ontology-guided closed extraction on the full French maintenance corpus—yields ontology-grounded legal knowledge graphs that achieve 100 percent JSON validity, near-complete class alignment, substantial reduction of duplicate entities and predicates after fusion, and property reuse above 80 percent (up to 96 percent after fusion), while lower exact signature compliance mainly signals new domain–range uses of existing predicates.

What carries the argument

The two-stage workflow itself: open class-guided triple extraction on a stratified sample, embedding-based label fusion at cosine threshold 0.7, induction of object properties with signatures, then closed signature-constrained extraction over the full corpus lifted to RDF via RML mappings and reified statements.

Load-bearing premise

That embedding fusion at a fixed similarity threshold plus prompt-level signature guidance, without a formal validation or repair loop or human gold triples, is enough to keep the extracted legal relations semantically faithful.

What would settle it

A human-annotated gold set of triples on a held-out sample of the same articles would show whether the fused, ontology-guided graphs match expert legal readings on entity types, predicates, and domain–range pairs, or whether structural metrics hide systematic meaning errors.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Predicate normalization and validation of newly observed domain–range signatures become the priority refinement steps for industrial use.
  • The same sample-then-guide pattern can be reused to keep the knowledge graph current when legal provisions change.
  • Reified RDF statements with evidence links support downstream GraphRAG-style consultation of maintenance obligations.
  • Different LLMs produce usable but distinct ontology variants (broader vs more conservative property sets) that can be compared for operational fit.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Without a closed validation loop, residual signature non-compliance may systematically encode entity-typing mistakes that propagate into operational risk assessments.
  • The sharp drop in entity and predicate counts after fusion suggests the main industrial value may lie more in normalization than in raw extraction volume.
  • Competency-question SPARQL tests already sketched in the paper could be turned into a lightweight regression suite for each legal update cycle.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a two-stage LLM-assisted pipeline for French maintenance regulations: (1) ontology engineering, in which open triple extraction on a stratified sample of 1,389 Légifrance articles is followed by embedding-based fusion of entity/property labels (cosine threshold θ_E = θ_P = 0.7) and induction of candidate object properties with domain–range signatures, yielding two SEMLEG-derived ontology variants (OpenAI: 75 properties/105 signatures; Mistral: 44/59); and (2) closed, signature-guided extraction over the full 6,370-article corpus, lifted to reified RDF (rdf:Statement / semleg:ExtractedRelation) via RML. Evaluation is via four structural metrics (R_JSON, R_class, R_prop, R_sig), a spot-check of inconsistent signatures, and competency-question SPARQL tests. Headline results: 100% JSON validity, ≥99.97% class alignment, R_prop up to 96.42% after Mistral fusion, R_sig 50–73%, and large post-fusion compression (e.g., Mistral entities 74,035→20,827; properties 2,643→500).

Significance. If the results hold, the paper contributes a compact, reproducible end-to-end recipe (prompts, code, and datasets are publicly released — a genuine strength) for moving from French legal text to an ontology-grounded, reified KG with provenance and evidence annotations, a combination few dataset-driven legal IE approaches provide. The separation of open induction from closed population, the honest reporting of low exact-signature compliance as an ontology-refinement signal rather than hiding it, and the acknowledgment that no formal validation/repair loop exists are commendable. The setting is a poster/demo track, so a preliminary evaluation is in scope; but the claims of a "usable" legal KG rest entirely on structural proxies, which limits the weight the current evidence can bear.

major comments (3)
  1. [§2 Embedding-Based Fusion; §4 Qualitative Analysis; Tables 2–3] Embedding fusion can merge predicates of opposite polarity, and no audit of fused clusters is reported. §2 defines fusion purely by cosine similarity (>0.7) on label embeddings, with canonical label chosen by frequency. Generic sentence embeddings assign very high similarity to strings differing only by negation or modality (e.g., 'mustExceed' vs 'mustNotExceed', 'appliesTo' vs 'cannotApplyFor'), which share nearly all tokens. The authors themselves state in §4 that both models 'encode modality or polarity directly in predicates' and warn that over-normalization 'can hide distinctions that are critical in industrial maintenance.' Yet Table 2 reports the collapse of 2,643 Mistral property variants to 500 (and 74,035 entities to 20,827) as an unqualified deduplication gain, and Table 3's post-fusion R_prop rise (82.51%→96.42%) will mechanically increase whenever fusion forces triples onto
  2. [§4 Quantitative Analysis (Table 3); §3] All four evaluation metrics are structural and partially self-referential; there is no human gold-standard check of triple correctness. R_prop is measured against an ontology whose properties were induced from triples extracted (and fused) by the same models on a sample of the same corpus, so high R_prop partly reflects that the ontology was built to fit the extraction distribution rather than that extracted relations are legally correct. Nothing in §4 verifies that any triple's content matches the source article. Given the paper's claim of a usable legal KG, at least a small-scale human evaluation is needed: e.g., annotate a random sample of ~100–200 rdf:Statement instances (stratified by fused/unfused) for correctness against the cited article text, and report precision. The competency-question SPARQL tests are a good start but only demonstrate retrieval, not correctness of what is ret
  3. [§3, Table 2] Table 2, '# Classes' row: fusion reduces the class count (Mistral 18→15, OpenAI 15→12). Entity fusion is described (§2) as merging labels of instances within each class, which should not change the number of instantiated classes. Either some classes lose all their instances through fusion (which would be surprising and worth explaining), or class typing is being altered as a side effect of entity merging, which interacts with Major Comment 1 (typing errors are already cited as the main source of inconsistent signatures). Please clarify what causes the class-count change and confirm it is not a symptom of unintended merges.
minor comments (5)
  1. [§4 Qualitative Analysis] The qualitative analysis reports 39 (Mistral) and 52 (OpenAI) inconsistent signatures but gives no denominator, so the reader cannot compute an inconsistency rate. Please state the total number of distinct fused signatures checked (presumably 1,636 and 3,398 from Table 2, or a subset) and clarify the selection procedure.
  2. [§2 Semantic Scope; Object Property Induction] The sampling parameters (10% stratified sample for ontology engineering, 5% of triples per property for induction, 15 batches) are given without sensitivity analysis or justification. A brief note on robustness to these choices would strengthen §2.
  3. [§4 Qualitative Analysis] §4 mentions 'occasionally generate French predicates despite prompts explicitly requiring English labels' — worth quantifying (even approximately), since language mixing affects the embedding-based fusion quality.
  4. [§2; Declaration on Generative AI] Typo: 'OpenaAI' (§2, paragraph after Table 1). Also the acknowledgment line 'takes full responsibility' should read 'take full responsibility'.
  5. [§4 Experimental Setup] Datasets are shared via a Google Drive link; for archival stability, a Zenodo/DOI deposit (as done for the GitHub code) is recommended.

Circularity Check

0 steps flagged

No significant circularity: standard ontology-induction-then-population pipeline; R_prop is an explicit coverage metric, not a fitted quantity renamed as prediction.

full rationale

This is an engineering systems paper, not a first-principles derivation. Stage 1 openly extracts triples from a stratified sample, fuses labels, and induces candidate object properties; Stage 2 uses that ontology to guide closed extraction on the full corpus and reports structural metrics (JSON validity, class coverage, property reuse R_prop, signature compliance R_sig). Measuring R_prop—whether closed triples reuse induced properties—is the intended success criterion of ontology-guided extraction, not a hidden fit presented as an independent prediction. Fusion thresholds (θ_E = θ_P = 0.7) are stated hyperparameters, not parameters fitted to a target and then “predicted.” Self-citations to SEMLEG [3,4] supply the retained core classes (Actor, Action, Artifact, …); that is ordinary reuse of prior infrastructure by overlapping authors and does not force the reported LLM extraction or fusion numbers. The paper itself flags lower R_sig and polarity-encoding predicates as open refinement issues rather than claiming uniqueness or completeness by construction. No equation equates an output to its input; no uniqueness theorem is imported; no ansatz is smuggled in as a theorem. Residual score 1 only for the mild, expected bootstrap of sample-induced vocabulary later scored by reuse—not definitional circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central systems claim rests on a small set of hand-chosen engineering thresholds, the adequacy of SEMLEG classes for maintenance obligations, and the untested premise that embedding synonymy plus LLM prompts preserve legal meaning. No physical constants; free parameters are pipeline knobs. Invented entities are the induced ontology variants and the populated graph, which have only internal structural evidence.

free parameters (4)
  • entity_label_fusion_threshold_theta_E = 0.7
    Cosine similarity cutoff for merging entity labels within each class; chosen as 0.7 without sensitivity study. Directly controls duplicate reduction and possible over-merge of distinct legal entities.
  • property_label_fusion_threshold_theta_P = 0.7
    Same 0.7 cutoff for merging property labels within (domain, range) class pairs; drives R_prop improvement and predicate vocabulary size.
  • ontology_induction_triple_sample_rate = 5% per property, 15 batches
    5% of fused maintenance triples per property (at least one per signature), split into 15 batches for relation-discovery prompts; controls which candidate properties are proposed.
  • stratified_corpus_sample_for_ontology_engineering = 1389 articles (~10% stratified)
    ~10% of each domain–document title pair (1,389 of 6,370 articles) used for open extraction and property induction; may miss rare relation types.
axioms (5)
  • domain assumption SEMLEG core classes (Actor, Action, Artifact, Condition, Source, Location, Reason, Situation, Time) and their object properties are a sufficient semantic skeleton for French maintenance obligations.
    Stated in §2 Semantic Scope; all typing and signature metrics are defined relative to this retained core.
  • ad hoc to paper Cosine similarity of embedding vectors above 0.7 identifies labels that are safe to collapse to a single canonical legal entity or predicate.
    §2 Embedding-Based Fusion; no legal synonymy gold set or threshold sweep is provided.
  • ad hoc to paper Prompt-injected admissible properties and signatures guide closed extraction well enough without formal SHACL-style validation or repair.
    Explicit in §3: constraints ‘are not enforced through a formal validation or repair loop’; future work proposes SHACL.
  • domain assumption Discarding triples labeled anotherLegalActivity or legalCrossReference does not remove maintenance-relevant structure needed for ontology induction.
    §2 retains only maintenanceActivity triples for T_maint before fusion and property induction.
  • ad hoc to paper Structural metrics (R_JSON, R_class, R_prop, R_sig) plus limited qualitative signature inspection are adequate to evaluate ontology-grounded KG quality for this study.
    §4 defines only these metrics; no human precision/recall on triples.
invented entities (2)
  • SemLegM ontology variants (OpenAI and Mistral maintenance-specific property sets) no independent evidence
    purpose: Extend SEMLEG with induced maintenance object properties and signatures to guide closed KG population.
    Created in §2 via LLM relation-discovery prompts; 75 vs 44 properties, 21 shared. No external ontology alignment study beyond internal overlap counts.
  • French maintenance legal knowledge graph (reified ExtractedRelation RDF graphs) no independent evidence
    purpose: Populate ontology-grounded triples over the full Légifrance maintenance corpus for query and future GraphRAG use.
    Constructed in §3; size statistics in Table 2. Evidence is internal structural metrics only; no external benchmark task scores.

pith-pipeline@v1.2.0-grok45-kimik3 · 11005 in / 3831 out tokens · 79502 ms · 2026-07-31T11:49:07.234636+00:00 · methodology

0 comments
read the original abstract

Maintenance regulations are complex legal texts that are difficult to exploit when addressing a specific case and challenging to integrate into operational systems. This paper presents a two-stage LLM-assisted workflow for French maintenance regulations: ontology engineering from a SEMLEG-based core ontology, followed by construction of an ontology-grounded French legal knowledge graph. The first stage consists in the open extraction of typed entities and triples from a stratified corpus sample, the normalization of labels through embedding-based fusion, and the induction of candidate object properties with their signature (domain and range). The second stage uses the resulting ontology to guide the closed extraction of triples and RDF graph construction over the full corpus. Experiments with GPT-4.1 and mistral-large-2512 show robust structured outputs, near-complete class alignment, and a substantial reduction of duplicated entities and predicates after fusion. Fewer than 20% of triples introduce unseen properties, while lower exact signature compliance reveals new domain-range combinations for existing predicates. These results point to predicate normalization and the validation of newly observed relation signatures as key refinement steps for industrial maintenance settings.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    Premasiri, T

    D. Premasiri, T. Ranasinghe, R. Mitkov, M. El-Haj, I. Frommholz, Survey on legal information extraction: Current status and open challenges, Knowledge and Information Systems 67 (2025) 11287–11358. doi:10.1007/s10115-025-02600-5

    work page doi:10.1007/s10115-025-02600-5 2025

  2. [2]

    S. Li, L. Yi, A few-shot entity relation extraction method in the legal domain based on large language models, in: Proceedings of the 2024 Guangdong-Hong Kong-Macao Greater Bay Area International Conference on Digital Economy and Artificial Intelligence, ACM, Hong Kong, China, 2024, pp. 580–586. doi:10.1145/3675417.3675513

    arXiv 2024

  3. [3]

    Breton, M

    J. Breton, M. B. Billami, M. Chevalier, C. Trojahn, Semantic model for the legal maintenance: The case of semantic annotation of france legislative and regulatory texts, in: Workshop on Methodolo- gies for Translating Legal Norms into Formal Representations (LN2FR 2022) in conjunction with JURIX 2022, Saarbrücken, Germany, 2022. URL: https://ut3-toulousei...

    2022

  4. [4]

    Breton, M

    J. Breton, M. B. Billami, M. Chevalier, C. Trojahn, Leveraging semantic model and LLM for bootstrapping a legal entity extraction: An industrial use case, in: Knowledge Graphs in the Age of Language Models and Neuro-Symbolic AI: Proceedings of the 20th International Conference on Semantic Systems, 17–19 September 2024, Amsterdam, The Netherlands, Studies ...

    work page doi:10.3233/ssw240004 2024

  5. [5]

    Woods, M

    C. Woods, M. Selway, T. Bikaun, M. Stumptner, M. Hodkiewicz, An ontology for maintenance activities and its application to data quality, Semantic Web 15 (2024) 319–352. doi: 10.3233/ SW-233299

    2024

  6. [6]

    V. A. Carriero, M. Scrocca, I. Baroni, A. Azzini, I. Celino, Procedural knowledge ontology (PKO),

  7. [7]

    Hodkiewicz, C

    M. Hodkiewicz, C. Woods, M. Selway, M. Stumptner, IOF-Maint: Modular maintenance ontology,

  8. [8]

    Ringwald, F

    C. Ringwald, F. Gandon, C. Faron, F. Michel, H. A. Akl, 12 shades of RDF: Impact of syntaxes on data extraction with language models, in: Lecture Notes in Computer Science, Lecture Notes in Computer Science, Springer Nature Switzerland, Hersonissos, Greece, 2024, pp. 81–91. doi: 10. 1007/978-3-031-78952-6_8

    2024

  9. [9]

    iText2KG: Incremental Knowledge Graphs Construction Using Large Language Models

    Y. Lairgi, L. Moncla, R. Cazabet, K. Benabdeslem, P. Cléau, iText2KG: Incremental knowledge graphs construction using large language models, 2024. URL: http://arxiv.org/abs/2409.03284. doi:10.48550/arXiv.2409.03284

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2409.03284 2024

  10. [10]

    Lairgi, L

    Y. Lairgi, L. Moncla, K. Benabdeslem, R. Cazabet, P. Cléau, ATOM: Adaptive and optimized dynamic temporal knowledge graph construction using LLMs, 2025. URL: http://arxiv.org/abs/2510.22590. doi:10.48550/arXiv.2510.22590

    work page doi:10.48550/arxiv.2510.22590 2025

  11. [2024]

    doi:10.26182/chzp-vs60

    URL: http://arxiv.org/abs/2404.05224. doi:10.26182/chzp-vs60

    Pith/arXiv arXiv

  12. [2025]

    doi:10.1007/978-3-031-94578-6_19

    URL: https://arxiv.org/abs/2503.20634v1. doi:10.1007/978-3-031-94578-6_19

    arXiv