Pith. sign in

REVIEW 3 major objections 4 minor 18 references

From Instructions to ODRL Usage Policies: An Ontology Guided Approach

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM guided by a distilled ODRL ontology and self-correction rules can turn natural-language usage instructions into ODRL policy knowledge graphs with up to 91.95% accuracy on the authors' benchmark.

desk verdict Useful practical recipe for LLM-to-ODRL generation, but the headline accuracy number measures conformance to the authors' own SHACL criteria rather than ODRL validity. read the letter →

arxiv 2506.03301 v1 pith:I4GXCPAI submitted 2025-06-03 cs.CL

Daham M. Mustafa , Abhishek Nadgeri , Diego Collarana , Benedikt T. Arnold , Christoph Quix , Christoph Lange , Stefan Decker This is my paper
classification cs.CL
keywords ODRLknowledgegraphconstructionlargelanguagemodelsontology-guidedgenerationself-correctionSHACLvalidationdataspacesusagepolicies
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a large language model can turn ordinary sentences about who may use what cultural data, where, and until when into machine-readable usage policies in ODRL, the W3C standard for digital rights, without a human ODRL expert in the loop. The authors argue that the decisive ingredient is not the raw ontology but a curated, distilled text version of it that explains ODRL syntax, semantics, and worked examples, followed by a second LLM pass that applies human-readable correction rules. On a benchmark of twelve cultural-domain dataspace use cases assessed by SHACL shapes, the refined pipeline reached up to 91.95% accuracy with GPT-4. If correct, a non-expert can produce ODRL policies that satisfy the structural checks used by dataspace infrastructures.

What carries the argument

The LLM Guidance Template is the central mechanism: a prompt built from the ODRL 2.2 ontology plus a distilled textual layer called OSES Insights, which reorganizes the specification into syntactic guidance, semantic distinctions (such as duty as obligation versus permission precondition), and worked policy examples. The second load-bearing component is the ODRL Self-Correction Rules, a set of 175 human-readable rules that the LLM applies to its first-draft knowledge graph. The evaluation machinery consists of SHACL shapes that encode criteria C1 through C9 across policy, permission, party, asset, and constraint shapes.

What would settle it

Have independent ODRL experts, blind to the authors' shapes, validate the twelve generated policies against the W3C ODRL recommendation; if a large fraction of graphs that pass the curated SHACL shapes fail expert review, the reported accuracy overstates real policy correctness.

Watch

Extended reading notes

Core claim

The central claim is that ontology-guided prompting, rather than the raw ODRL ontology alone, drives accurate knowledge-graph construction. The authors report that translating the ODRL ontology into plain text organized as syntax, semantics, and examples substantially improves the generated graphs, and that a subsequent self-correction pass using 175 hand-written rules improves accuracy further. In their evaluation, the best refined run reaches 91.95% accuracy, measured as the fraction of satisfied checks in SHACL shapes encoding nine ODRL criteria, and the refinement condition consistently outperforms the ontology-only condition across the tested models.

Load-bearing premise

The load-bearing premise is that the manually written criteria C1-C9 and their SHACL encodings are a correct and sufficient gold standard for ODRL policy validity, so passing those shapes on a single run is taken as proof that the policy is right.

Editorial extensions

If this is right

  • A domain expert who can state a usage policy in natural language can generate a policy knowledge graph without first learning RDF, Turtle, or the ODRL vocabulary.
  • The recipe of distilling an ontology into plain text, adding worked examples, and asking an LLM to self-correct against human-readable rules may transfer to other ontology-driven knowledge-graph construction tasks.
  • Dataspace components that consume ODRL policies can ingest the generated graphs, since the outputs pass the SHACL validation used in the paper.
  • Self-correction against textual rules outperforms feeding SHACL violation messages back to the LLM, because the LLM can edit the graph rather than only report violations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported accuracy is measured against SHACL shapes the authors constructed; an independent gold standard might score the same graphs lower, so the headline number is best read as an upper bound on policy validity.
  • Because the distilled guidance and correction rules were created by manual interpretation of the W3C recommendation, the method's ceiling is likely set by the faithfulness of that hand-curated layer, not by the LLM alone.
  • A natural next test is multiple runs at varied temperature on the same use cases; the paper's single-run evaluation leaves run-to-run variation unmeasured, and borderline policies may flip between valid and invalid across runs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an end-to-end LLM pipeline for generating ODRL usage-policy knowledge graphs from natural language instructions. The pipeline builds LLM guidance templates from the ODRL ontology, adds distilled textual "OSES" insights with syntax, semantics, and examples, and then applies a self-correction step driven by 175 hand-written rules. Twelve cultural-domain use cases are scored against nine criteria C1-C9 encoded as manually created SHACL shapes. The reported headline result is "up to 91.95% accuracy," obtained for Use Case 1 after refinement with GPT-4-class models.

Significance. If the accuracy claim were robust, the approach would be practically valuable: it would let non-ODRL experts generate syntactically structured ODRL policies from natural language, lowering an entry barrier for dataspaces. The paper has concrete strengths: it ships the dataset, prompts, and code in a public repository; it compares three prompt/guidance conditions (ontology-only, OSES insights, and refinement); and it uses SHACL-based scoring in a repeatable way. The authors also explicitly acknowledge in Section 5 that the guidance templates and self-correction rules are derived by manual interpretation of the ODRL specification. The main weakness is that the evaluation cannot yet establish external ODRL correctness because the scoring shapes encode the authors' own interpretation and the reported accuracy is a single best-case run.

major comments (3)
  1. [§4, Table 1 (C2, C3); §4 Criteria Definition] The SHACL gold standard is not an independent ODRL validity criterion. Table 1 C2 requires explicit XSD datatype specification for all values, and C3 requires every policy to include dc:creator, dc:title, dc:description, and dc:issued. ODRL 2.2 does not mandate Dublin Core metadata, and it does not require every literal to carry an explicit datatype annotation (RDF 1.1 supplies xsd:string as the default datatype for simple literals). Section 4 states that each criterion was translated into SHACL constraints "manually generated for this study." Therefore the computed accuracy partly measures adherence to the authors' project-specific conventions rather than conformance to the W3C ODRL specification. Please score against an independent validator or against shapes restricted to ODRL-mandated constraints, and report scores for spec-mandated and project-specific criteria separately.
  2. [§4, Results and Eq. (1)] The headline accuracy is a single best-case run, not a stable estimate. Section 4 reports "Initially, we ran each use case once," and the 91.95% figure is derived as 217/236 for Use Case 1 with refinement. Equation (1) computes an R/T ratio, but no confidence intervals, standard deviations, or repeated runs are provided for a stochastic LLM. Reporting "up to 91.95%" is a maximum over cases, not an average, and it does not support the claim of "excellent results" across the benchmark. Please report mean and median accuracy across all 12 use cases with multiple independent runs (e.g., different temperatures or seeds) and provide variance information or paired comparisons across the three conditions.
  3. [§3.3, §4, §5] There is a circularity risk between the self-correction rules and the evaluation criteria. The 175 self-correction rules in Section 3.3 and the SHACL shapes in Section 4 are both manually derived from the same W3C ODRL specification by the same authors, and Section 5 explicitly acknowledges that the LGT guidelines and self-correction rules involve manual interpretation and analysis. The refinement gain therefore partly measures how well the LLM follows the authors' hand-authored conventions, not how well the output conforms to an external standard. To separate these effects, the evaluation should include an independent gold standard: for example, validation by independent ODRL experts with inter-annotator agreement, or a comparison against an independent ODRL validator, or an ablation in which self-correction is driven only by the standard SHACL shapes rather than the authors' rules.
minor comments (4)
  1. [§1 and Figure 1] The text says Figure 1 shows a high-level overview of the ODRL information model, but the figure appears to depict the proposed method with a painting example; the workflow figure is Figure 2. Please correct the cross-reference or the figure content.
  2. [Eq. (1)] The terms R and T in the accuracy formula are not defined precisely enough; please state which shapes and focus nodes are included in the total possible score and how partial credit is assigned.
  3. [Table 1, C9] The phrase "adhere to the the ODRL Profile Mechanism" contains a duplicated definite article.
  4. [Figure 3 caption] The caption says the LLMs "learn progressively from ontology input," but no model training or weight updates occur in the experiments; this wording should be replaced with something like "performance improves as the guidance is augmented."

Circularity Check

2 steps flagged · score 6.0 of 10

The 91.95% headline accuracy is scored against hand-authored SHACL shapes that are also given to the LLM as part of the prompt, so the central correctness claim measures self-consistency with the authors' own ODRL interpretation rather than independent ODRL validity.

  1. self definitional [Section 4 (Evaluation, Criteria Definition and Assignment of Scores) and Figure 1 caption]
    "Each criterion is translated into a SHACL shape with associated constraints manually generated for this study. ... Using an LLM, a curated description of the ODRL ontology and associated SHACL shapes, we generate a KG that corresponds to the ODRL representation of the described policy."

    The same SHACL shapes appear on both sides of the derivation: Figure 1 includes them in the LLM input, while Section 4 uses them as the scoring rubric behind Accuracy = ΣR/ΣT × 100%. The headline 91.95% is therefore not a check against an external ODRL gold standard but a measure of how faithfully the LLM followed the authors' own hand-written shapes. This is made worse by non-ODRL requirements in the rubric: C2 mandates explicit XSD datatypes and C3 mandates dc:creator, dc:title, dc:description, and dc:issued, none of which ODRL 2.2 requires. A policy that is valid ODRL but lacks these author-added features is penalized by the same shapes that guided generation, so the metric is self-referential by construction.

  2. other [Section 3.3 (ODRL Self-Correction Model) and Section 5 (Conclusions)]
    "These rules, termed correction rules, are derived from the ODRL W3C recommendation and ontology relations, ensuring alignment with the official ODRL specification. ... In our current methodology, LGT guidelines for KG construction (section 3.1) and selfcorrection rules (section 3.3) have been derived and formulated from the W3C Recommendation and ODRL ontology, which involves manual interpretation and analysis."

    The best-scoring outputs come from applying the hand-written self-correction rules, and the SHACL evaluation criteria C1-C9 were produced by the same authors' manual interpretation of the same W3C text. The paper explicitly admits this manual-interpretation loop. Thus the refinement gain is not independent evidence that the pipeline approaches ODRL correctness defined outside the authors' reading; it shows that the LLM can be steered toward the exact interpretation encoded in the test rubric. No numerical parameter is fitted, so this is partial rather than total circularity, but the input rules and the oracle share a single human source.

full rationale

The claimed derivation chain is: curated ODRL ontology + distilled insights + self-correction rules → ODRL KG → accuracy. The accuracy is scored with SHACL shapes that the paper says were manually generated for this study, and Figure 1 indicates those same shapes are part of the LLM input. Moreover, Section 5 concedes that both the LGT guidelines and the self-correction rules were manually derived from the W3C Recommendation and ODRL ontology. The W3C ODRL specification is a real external standard, so this is not a pure tautology and there are no fitted numerical parameters or load-bearing self-citations. Nevertheless, the absolute headline claim of up to 91.95% accuracy reduces to the rate at which the LLM reproduces the authors' hand-authored shapes and rules. The relative improvements of OSES Insights and Refinement might survive under any fixed rubric, but the paper's central 'ODRL correctness' result is not anchored to an independent oracle. This warrants a partial-circularity score of 6.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric free parameters were fitted to data; the approach relies on hand-crafted prompts, human-readable rules, and manually built SHACL shapes. The recorded axioms are the load-bearing assumptions about the authority of the ODRL spec, the validity of the authors' evaluation shapes, the representativeness of single LLM runs, and the generality of the 12-case benchmark.

assumptions (4)
  • domain assumption The W3C ODRL 2.2 ontology and recommendation are a correct and sufficient specification of usage policy semantics.
    The LGT, the self-correction rules, and criteria C1-C9 are all constructed from these sources in Sections 3.1-3.3.
  • ad hoc to paper The manually generated SHACL shapes faithfully encode the intended ODRL validity criteria.
    Section 4 states each criterion is translated into a SHACL shape manually generated for this study; no independent validation of the shapes is provided.
  • ad hoc to paper A single run of a stochastic LLM is representative of its performance on a task.
    The evaluation reports one run per model-method-use case combination in Section 4, ignoring sampling variance.
  • domain assumption Twelve use cases from one cultural-domain project generalize to ODRL policy generation more broadly.
    The dataset is described in Section 4 as derived from the Datenraum Kultur project; no external benchmark is used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Instructions to ODRL Usage Policies: An Ontology Guided Approach." pith.science (2026). https://pith.science/paper/I4GXCPAI

@misc{pith2026250603301,
  author       = {Pith},
  title        = {Pith review of: From Instructions to ODRL Usage Policies: An Ontology Guided Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I4GXCPAI}},
  note         = {Machine review of arXiv:2506.03301}
}
read the original abstract

This study presents an approach that uses large language models such as GPT-4 to generate usage policies in the W3C Open Digital Rights Language ODRL automatically from natural language instructions. Our approach uses the ODRL ontology and its documentation as a central part of the prompt. Our research hypothesis is that a curated version of existing ontology documentation will better guide policy generation. We present various heuristics for adapting the ODRL ontology and its documentation to guide an end-to-end KG construction process. We evaluate our approach in the context of dataspaces, i.e., distributed infrastructures for trustworthy data exchange between multiple participating organizations for the cultural domain. We created a benchmark consisting of 12 use cases of varying complexity. Our evaluation shows excellent results with up to 91.95% accuracy in the resulting knowledge graph.

Figures

Figures reproduced from arXiv: 2506.03301 by the authors.

Figure 1
Figure 1. Our method takes a natural language description of a usage policy as input. Using an LLM, a curated description of the ODRL ontology and associated SHACL shapes, we generate a KG that corresponds to the ODRL representation of the described policy. within Germany but prohibited within the USA”, a domain expert is required to know all the terminology from ODRL, including Asset, Permission, Constraint, etc. The ODRL ve… view at source ↗
Figure 2
Figure 2. Approach. The LLM Guidance Template (LGT) comprises two modules, i.e., the ODRL ontology in Turtle serialization and contextual insights in PDF format. KG is constructed by passing Task Description (TD) and the LGT as input to the LLM. Next, the Self-correction Model (SCM ) refines the generated KG by providing the TD, the first version of the KG, and the ODRL Self-Correction Rules (SCR) as input for the LLM to prod… view at source ↗
Figure 3
Figure 3. However, in the 𝑂𝑆𝐸𝑆 Insights approach, the ODRL 𝐾𝐺 is enhanced with prop￾erties and relations. By formulating the ontology in text and providing semantic, syntactic, and real examples, the guidance provided to the LLM is improved. Refined ODRL KG quality improves through self-correction rules11 targeting crucial omitted properties. This iterative 6 https://datenraum-kultur.fit.fraunhofer.de/ 7The use cases can be f… view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 12 canonical work pages

  1. [1]

    T. Dam, A. Krimbacher, S. Neumaier, Policy patterns for usage control in data spaces, arXiv preprint arXiv:2309.11289 (2023)

    work page Pith review arXiv 2023

  2. [2]

    Villata, R

    S. Villata, R. Iannella, ODRL Information Model 2.2, Recommendation REC-odrl-model- 20180215, W3C, 2018. URL: https://www.w3.org/TR/2018/REC-odrl-model-20180215/

    work page 2018

  3. [3]

    Knublauch, D

    H. Knublauch, D. Kontokostas, Shapes Constraint Language (SHACL), Recommendation REC-shacl-20170720, W3C, 2017. URL: https://www.w3.org/TR/2017/REC-shacl-20170720/

    work page 2017

  4. [4]

    Y. Liu, Y. Wan, L. He, H. Peng, S. Y. Philip, Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021, pp. 6418–6425

    work page 2021

  5. [5]

    H. Ye, N. Zhang, H. Chen, H. Chen, Generative knowledge graph construction: A review, in: Y. Goldberg, Z. Kozareva, Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022. URL: https://aclanthology.org/2022.emnlp-main.1. doi:1...

    doi:10.18653/v1/2022.emnlp-main.1 2022

  6. [6]

    Keraghel, S

    I. Keraghel, S. Morbieu, M. Nadif, A survey on recent advances in named entity recog- nition, CoRR abs/2401.10825 (2024). URL: https://doi.org/10.48550/arXiv.2401.10825. doi:10.48550/ARXIV.2401.10825. arXiv:2401.10825

    arXiv doi:10.48550/arxiv.2401.10825 2024

  7. [7]

    Xiaoyan, D

    Z. Xiaoyan, D. Yang, Y. Min, W. Lingzhi, Z. Rui, C. Hong, L. Wai, S. Ying, X. Ruifeng, A comprehensive survey on deep learning for relation extraction: Recent advances and new frontiers, arXiv preprint arXiv:2306.02051 (2023)

    arXiv 2023

  8. [8]

    Sevgili, A

    Ö. Sevgili, A. Shelmanov, M. Arkhipov, A. Panchenko, C. Biemann, Neural entity linking: A survey of models based on deep learning, Semantic Web 13 (2022) 527–570

    work page 2022

Show all 18 references
  1. [9]

    R. Liu, R. Mao, A. T. Luu, E. Cambria, A brief survey on recent advances in coreference resolution, Artificial Intelligence Review (2023) 1–43

    2023

  2. [10]

    Hofer, D

    M. Hofer, D. Obraczka, A. Saeedi, H. Köpcke, E. Rahm, Construction of knowledge graphs: State and challenges, arXiv preprint arXiv:2302.11509 (2023)

    2023 arXiv

  3. [11]

    Y. Zhu, X. Wang, J. Chen, S. Qiao, Y. Ou, Y. Yao, S. Deng, H. Chen, N. Zhang, Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities, arXiv preprint arXiv:2305.13168 (2023)

    2023 arXiv

  4. [12]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., On the opportunities and risks of foundation models, arXiv preprint arXiv:2108.07258 (2021)

    2021 arXiv

  5. [13]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, Google, kt, language, ai: Bert: pre-training of deep bidirectional transformers for language understanding, in: Proceedings of NAACL-HLT, 2019, pp. 4171–4186

    2019

  6. [14]

    Kumar, A

    A. Kumar, A. Pandey, R. Gadia, M. Mishra, Building knowledge graph using pre-trained language model for learning entity-aware relationships, in: 2020 IEEE International Conference on Computing, Power and Communication Technologies (GUCON), IEEE, 2020, pp. 310–315

    2020

  7. [15]

    Q. Guo, Y. Sun, G. Liu, Z. Wang, Z. Ji, Y. Shen, X. Wang, Constructing chinese historical literature knowledge graph based on bert, in: Web Information Systems and Applica- tions: 18th International Conference, WISA 2021, Kaifeng, China, September 24–26, 2021, Proceedings 18, ...

    2021

  8. [16]

    J. Han, N. Collier, W. Buntine, E. Shareghi, Pive: Prompting with iterative verification improving graph-based generative capability of llms, arXiv preprint arXiv:2305.12392 (2023)

    2023 arXiv

  9. [17]

    T. R. Gruber, A translation approach to portable ontology specifications, Knowledge acquisition 5 (1993) 199–220

    1993

  10. [18]

    Anand, Z

    Y. Anand, Z. Nussbaum, B. Duderstadt, B. Schmidt, A. Mulyar, Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo, GitHub (2023)

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.