REVIEW 4 major objections 6 minor 15 references
Ontology-Based Structuring and Analysis of North Macedonian Public Procurement Contracts
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A knowledge-graph pipeline turns North Macedonia's procurement contracts into queryable, predictable data.
desk verdict A competent data-engineering case study with a weak, possibly in-sample ML evaluation; the knowledge-graph part is sound but the abstract's risk-assessment claim is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a small public procurement ontology: the Contract class linked to Institution and Supplier through the object properties hasInstitution and hasSupplier, with datatype properties hasAmount, hasDate, and hasDescription, enforced by OWL cardinality restrictions. This schema dictates how RML maps spreadsheet attributes to RDF, what SHACL shapes validate, and what SPARQL queries can express. For prediction, the mechanism is a nearest-neighbor estimator: contract descriptions are embedded, compared by cosine similarity against a pre-indexed corpus, and the median amount of the top-nine similar contracts becomes the value estimate.
What would settle it
Run the same prediction experiment with a documented leave-one-out split, where each test contract is removed from the similarity index before retrieving neighbors; if the reported RMSE of 39.92 and $R^2$ of 0.056 are reproduced only when test contracts remain in the index, the predictive claim fails.
Extended reading notes
Core claim
The central claim is that public procurement data need not remain trapped in rigid tabular files: by modeling contracts through a small OWL ontology, converting CSV records to RDF with RML, validating them with SHACL, and loading them into a triple store, the dataset becomes a knowledge graph that supports flexible semantic queries. On this graph, the paper executes trend queries over 896 contracts from 2009 to 2021, identifying annual totals, largest contracts, most active institutions, and dominant supplier pairs. The paper also claims that an embedding-based retrieval system, which encodes contract descriptions, finds the nine nearest historical contracts, and takes the median of their values, predicts contract amounts with RMSE 39.92, MAE 12.77, and $R^2$ 0.056, compared with the median baseline RMSE 42.21, MAE 13.23, and $R^2$ -0.057.
Load-bearing premise
The machine-learning result depends on each test contract being absent from the similarity index built from historical contracts, but the paper does not describe how training and test contracts were separated.
Editorial extensions
If this is right
- Anyone can query total public spending, top suppliers, and unusual contracts directly, without waiting for pre-built reports.
- The ontology's required relations, every contract must have a supplier and an institution, act as a data-quality check that can be run before publication.
- Embedding-based estimates give procurement planners a starting point for contract values before formal bids arrive.
- The twelve-year knowledge graph supports longitudinal studies of spending shifts across governments and economic cycles.
- Because the pipeline starts from XLSX files, it can be re-run whenever new data is published, keeping the graph current.
Reading between the lines
- Beyond the paper, the same ontology-and-SHACL workflow could be applied to other tabular public datasets such as budget lines, permits, or subsidies, giving each a queryable semantic layer without redesign.
- A natural extension is to expand the ontology to include tender stages such as announcements, bids, and awards, so that risk analysis can look at competition and bid spreads rather than only final contract values.
- The nearest-neighbor estimator suggests a testable pattern: if contract descriptions are informative, prediction error should shrink as the similarity index grows; this could be checked on the full low-value contract portal.
- The framework implicitly proposes that semantic validation and predictive analytics belong together, so a practical next step would be publishing SHACL validation reports alongside each procurement dataset.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an applied pipeline that aggregates North Macedonian high-value public procurement records (896 contracts, 2009–2021), maps them from CSV/XLSX to RDF using an OWL ontology and RML mappings, validates the resulting graph with SHACL, and runs descriptive SPARQL queries to report summary statistics and top institutions, suppliers, and contract values. It then proposes a contract-amount prediction approach based on multilingual sentence embeddings (multilingual-e5-large-instruct) and a FAISS k-nearest-neighbor search, reporting RMSE, MAE, and R2 against a median baseline. The abstract and conclusion frame the work as a framework for semantic procurement analytics that includes machine-learning-based forecasting and risk assessment.
Significance. If judged only as a data-engineering case study, the paper is a plausible and potentially useful demonstration: the RML/SHACL pipeline is standards-based, the SPARQL queries are clearly described, and the summary statistics address transparency-relevant questions. The predictive component, however, is a central element of the stated contribution and is not established: the evaluation lacks a specified train/test split, uses a small dataset, and reports R2=0.056, which is near zero; no risk-assessment or anomaly-detection analysis is actually carried out. Strengths of the paper are its use of public data and the relatively simple, reproducible structure of the ontology and queries; no code, ontology file, or SPARQL scripts are provided, which limits verification. As it stands, the paper can be accepted only as a descriptive case study after the ML claims are substantially revised or removed.
major comments (4)
- [§VII-A and VII-B] The evaluation protocol for the predictive model is underspecified. §VII-A states that 'all previously known procurement contracts from our historical dataset are pre-encoded and stored in a FAISS index' and that a query is posed for 'a new procurement contract', but the paper never states how a test contract is separated from the indexed contracts, nor how many folds or repetitions were used. With only 896 contracts, retrieving the test contract's own embedding as its nearest neighbor would materially inflate similarity and deflate error. The authors must specify a true out-of-sample scheme (e.g., leave-one-out cross-validation with the query excluded from the index) and report the resulting performance; without this, the RMSE/MAE improvements in Table III cannot be interpreted as evidence of generalization.
- [Table III and end of §VII-B] Even if the evaluation were out-of-sample, the reported result does not support the claim that 'the model captures key patterns in procurement data'. R2=0.056 means the model explains only about 5.6% of the variance in contract values, and the reductions relative to the median baseline are small (RMSE 42.21→39.92, MAE 13.23→12.77). The paper does not define how R2 is computed for a median baseline, nor does it provide confidence intervals, standard deviations, or a significance test. The near-zero R2 and modest error reduction should be described as a weak baseline comparison, not as evidence of predictive skill.
- [Abstract and §VIII] The abstract's claim that the system offers 'insights into procurement trends and risk assessment' and the conclusion's mention of 'anomaly detection' are not supported by the analysis presented. No risk model, risk indicator, anomaly-detection experiment, or threshold-based outlier analysis is defined in §VII. Predicting the monetary value of a contract via nearest neighbors is not equivalent to risk assessment. Either a concrete risk/anomaly analysis must be added, or the claims in the abstract and conclusion must be narrowed to contract-amount estimation and descriptive trend analysis.
- [§II-B, §VI, Table II] The dataset is said to contain 'high-value contracts exceeding 1,000,000 euros', but all monetary values reported in the paper are in MKD (e.g., 241,083,174,450 MKD in Table II) without a stated exchange rate or conversion. This unit ambiguity matters for the ML target as well: it is unclear whether the RMSE/MAE values in Table III are in millions of MKD and how the EUR threshold was applied. The paper should specify the currency conversion and clearly state that all conclusions apply only to the high-value stratum, which is a selected subset of procurement activity.
minor comments (6)
- [Table II] The entry '7,2205' for 'Average contracts per institution' appears to be a typo; 896/127 ≈ 7.06, and the decimal separator should be consistent (elsewhere dots are used).
- [References] References [12] (Jones, term specificity) and [13] (Breiman, random forests) are not used appropriately: [12] is not cited in the text, and [13] is cited in §VII-B although no random-forest model is employed.
- [Reproducibility] For reproducibility, the authors should provide a link to the ontology, the RML mapping files, the SHACL shapes, the SPARQL queries, and the prediction code, as well as the exact snapshot date of the open-data portal.
- [§V-A] The text says 'Unique Resource Identifiers' but the standard term is 'Uniform Resource Identifiers' (or IRIs); the paper should also describe the URI scheme used for institutions, suppliers, and contracts.
- [Figures 3 and 4] Figures 3 and 4 are referenced but not accompanied by the actual plots in the submitted manuscript; if they are available, they should be embedded, and the axes and units should be labeled.
- [§VII-A] The embedding model 'multilingual-e5-large-instruct' has no citation or version identifier; the paper should cite the model card or a corresponding publication, and report embedding dimensionality and the choice of k=9 nearest neighbors without sensitivity analysis.
Circularity Check
No significant circularity: the ontology/RDF/SPARQL pipeline is self-contained, the kNN prediction is non-parametric, and no load-bearing self-citation appears.
full rationale
The paper's derivation chain contains no step in which an input is defined in terms of an output, a fitted parameter is relabeled as a prediction, or a self-citation carries the argument. The ontology design in Section IV maps existing CSV fields (Contracting Authority, Procurement Holder, Contract Value, Date) into classes and properties; this is a standard modeling exercise, not a conclusion derived from the ontology itself. The RML conversion and SHACL validation in Section V are conventional transformations whose outputs are the RDF graph and validation reports, and nothing in those sections presupposes the paper's own later analytic claims. The SPARQL queries in Section VI simply aggregate the transformed data, so their results are factual outputs of the data, not circular confirmations of the framework. Section VII uses a transformer embedding model and a FAISS nearest-neighbor index to estimate contract values from the median of the top-9 similar contracts; this is a non-parametric estimator with no fitted parameter that could be said to encode the target values, so the prediction is not circular by construction. The references are to external standards, government portals, and standard ML/Semantic Web tooling, with no self-citation chain and no imported uniqueness theorem. The most significant weakness is that Section VII-A does not explicitly describe excluding each test contract from the FAISS index, leaving the out-of-sample validity of the reported metrics uncertain; however, this is an underspecified evaluation protocol, not a demonstrated equation-level reduction of the prediction to its input. Similarly, the low R² of 0.056 is a correctness or overclaiming concern, not circularity. Therefore, under the hard rule that circularity must be exhibited with a specific reduction, the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (3)
- k (number of nearest neighbors) =
9
- Embedding model =
multilingual-e5-large-instruct
- High-value contract threshold =
>1,000,000 EUR
assumptions (4)
- domain assumption The public EPPS dataset on the open data portal is accurate and complete for high-value contracts from 2009 to 2021.
- domain assumption The three-class ontology (Contract, Institution, Supplier) with hasAmount, hasDate, hasDescription adequately represents the procurement domain.
- domain assumption Cosine similarity between multilingual-e5 embeddings of contract descriptions is a meaningful proxy for similarity in contract value.
- domain assumption The FAISS index and median aggregation produce a valid estimator of contract value, without stated distributional assumptions.
Cite this review
Pith. "Pith review of Ontology-Based Structuring and Analysis of North Macedonian Public Procurement Contracts." pith.science (2026). https://pith.science/paper/725KUWJX
@misc{pith2026250509798,
author = {Pith},
title = {Pith review of: Ontology-Based Structuring and Analysis of North Macedonian Public Procurement Contracts},
year = {2026},
howpublished = {\url{https://pith.science/paper/725KUWJX}},
note = {Machine review of arXiv:2505.09798}
}
read the original abstract
Public procurement plays a critical role in government operations, ensuring the efficient allocation of resources and fostering economic growth. However, traditional procurement data is often stored in rigid, tabular formats, limiting its analytical potential and hindering transparency. This research presents a methodological framework for transforming structured procurement data into a semantic knowledge graph, leveraging ontological modeling and automated data transformation techniques. By integrating RDF and SPARQL-based querying, the system enhances the accessibility and interpretability of procurement records, enabling complex semantic queries and advanced analytics. Furthermore, by incorporating machine learning-driven predictive modeling, the system extends beyond conventional data analysis, offering insights into procurement trends and risk assessment. This work contributes to the broader field of public procurement intelligence by improving data transparency, supporting evidence-based decision-making, and enabling in-depth analysis of procurement activities in North Macedonia.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
- [1]
- [2]
- [3]
- [4]
-
[5]
Open Data Portal: Access to Government Procurement Data
-
[6]
Public Procurement Contracts Over 1 Million Euros
-
[7]
Noy F., McGuinness D.L., ”Ontology Development 101,” Stanford KSL Technical Report KSL-01-05
-
[8]
Hitzler P., et al., ”OWL 2 Web Ontology Language Primer” W3C
Show all 15 references
-
[9]
Dimou A., et al., ”RML: A Generic Language for RDF Mappings” LDOW 2014
2014
-
[10]
Knublauch H., Kontokostas D., ”SHACL-Shapes Constraint Language”
-
[11]
Harris S., Seaborne A., ”SPARQL 1.1 Query Language” W3C
-
[12]
Jones K.S., ”Statistical Interpretation of Term Specificity” Journal of Documentation
-
[13]
Breiman L., ”Random Forests” Machine Learning Journal
-
[14]
30, 2017
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I., ”Attention Is All You Need,” Advances in Neural Information Processing Systems (NeurIPS) , vol. 30, 2017
2017
-
[15]
Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazar ´e, P.-E., Lomeli, M., Hosseini, L., and J ´egou, H., ”The FAISS Library,” arXiv preprint arXiv:2401.08281 , 2024
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.