REVIEW 3 major objections 6 minor 41 references
Enhancing PyKEEN with Multiple Negative Sampling Solutions for Knowledge Graph Embedding Models
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that a modular PyKEEN extension can host five advanced negative samplers, and that link prediction performance stays nearly flat as the number of negatives per positive grows because small negative pools are topped up with
desk verdict Useful PyKEEN extension for negative samplers, but the headline empirical claim about extra negatives is an artifact of the random-fallback rule, and the paper should be revised to separate the tool contribution from the confounded experiment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the negative pool, $\mathcal{P}_{\text{head}}(\langle h,r,t\rangle)$ and $\mathcal{P}_{\text{tail}}(\langle h,r,t\rangle)$, the set of entities that can replace the head or tail without producing a triple already in the graph. Each sampler is one rule for building these pools; the extension's base class reduces implementing a sampler to a precompute method and a candidate-negatives method, while a supplementation parameter fills short pools with random entities. That last mechanism does most of the explanatory work in the empirical study.
What would settle it
Re-run the FB15K and WN18 link prediction experiments at negatives per positive 1, 2, 5, 20, 50, 100 with the supplementation parameter disabled, or with duplicates resampled from the restricted pool instead of random entities. If the Hits@10 curves separate by sampler or slope noticeably with the negative count, the paper's 'negligible impact' finding reflects the fallback rather than the samplers. The paper's own pool statistics indicate where to look: relational sampling has average pools of 2 to 7 entities, far below 100.
Extended reading notes
Core claim
The central claim is that a PyKEEN extension can provide a consistent, reusable home for advanced negative samplers, and that the experiments reveal a boundary condition for those samplers. Each sampler is defined by how it constructs the negative pools $\mathcal{P}_{\text{head}}(\langle h,r,t\rangle)$ and $\mathcal{P}_{\text{tail}}(\langle h,r,t\rangle)$ for a positive triple; static strategies precompute candidate sets from relational structure, types, or co-occurrence assumptions, while dynamic strategies use an auxiliary model's embeddings to pick hard negatives. The proof-of-concept study on FB15K and WN18 shows that raising the number of negatives per positive from 1 to 100 leaves Hits
Load-bearing premise
The headline empirical result rests on the rule that fills undersized negative pools with random entities; if that rule were changed, the measured flatness could disappear.
Editorial extensions
If this is right
- Practitioners can compare five negative samplers inside the standard PyKEEN pipeline without forking model code, and hyperparameter search can treat the sampler as just another configuration choice.
- On the tested datasets, sampler identity matters more than negative count: at 100 negatives per positive the behavior converges toward random corruption, so gains from 'more negatives' should not be expected beyond a small threshold.
- Typed and relational samplers are viable mainly on datasets with rich type metadata and simple relational patterns; their average pool sizes, as low as 2 entities for relational sampling, quantify this constraint.
- The pool-size statistics supply a reusable diagnostic for deciding in advance whether a dataset can support a given sampling strategy.
- Using RESCAL as the auxiliary model can drag down dynamic and adversarial samplers, so dynamic sampler comparisons are coupled to the choice of auxiliary model.
- Vanilla PyKEEN's basic negative sampler set is a real gap: the extension's standardized interface makes advanced strategies as easy to invoke as random corruption.
Reading between the lines
- A natural next experiment, not run in the paper: disable supplementation and observe whether advanced samplers outperform random at high negative counts; the pool statistics suggest they may saturate early instead.
- The extension's modular design invites combining schema constraints with a tunable random mixing fraction, which could let practitioners control the precision-coverage trade-off the paper exposes.
- The paper measures Hits@10 only; negative-sample quality metrics such as false-negative rates or pool purity would be needed to separate sampler quality from pool-size effects in these results.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a modular extension to the PyKEEN knowledge-graph embedding framework that implements several negative sampling strategies (Corrupt, Typed, Relational, Nearest Neighbor, Adversarial) on top of PyKEEN's existing random and Bernoulli samplers. The extension provides a common abstraction with two methods for samplers, a fallback mechanism for negative pools that are too small, a custom filterer for invalid triples, and a data loader for semantic metadata. As a proof of concept, the authors report negative-pool statistics on four datasets and filtered Hits@10 link prediction results on FB15K and WN18, with hyperparameter optimization and a sweep over the number of negatives per positive (1, 2, 5, 20, 50, 100). The main empirical observation is that increasing the number of negatives has negligible impact on performance, which the paper attributes to the random-supplementation rule kicking in when a sampler's pool is too small.
Significance. The software contribution is genuinely useful: a clean, documented, and extensible implementation of advanced negative samplers inside the popular PyKEEN ecosystem, with examples for training, hyperparameter optimization, and standalone use, would lower the barrier for practitioners and researchers to test a variety of sampling strategies. The architecture (two abstract methods, a fallback flag, a filter for invalid triples, and an external-metadata loader) is well conceived. The empirical value is currently limited because the headline 'negligible impact of more negatives' conclusion is confounded by the paper's own random-fallback rule, as the paper itself acknowledges. The resource paper itself is sound and worth publishing after experimental revision.
major comments (3)
- [§5.2, Table 3, §4.1] The claim that increasing the number of negatives per positive has negligible impact on Hits@10 is an artifact of the supplementation rule. For Relational sampling on WN18, the average pool is 2 and 99.9% of triples have fewer than 100 negatives; for Typed on YAGO4-20, 16% of triples have no pool at all. Thus for N=20, 50, 100, most negatives under Corrupt, Typed, and Relational are random fallback negatives, so the curves essentially compare random sampling at different N. The paper admits this in §5.2: 'the sampling behavior becoming increasingly similar to the random corruption scheme as the number of negatives increases.' To support the 'negligible impact' conclusion, the authors should disable supplementation (either by skipping triples with insufficient pools or by capping N at the 95th percentile of pool sizes) and rerun; it is quite possible that samplers diverge at high N when t
- [§4.1 and §5.2, Figs. 2-3] The comparative link prediction results lack statistical support: no repeated seeds, no confidence intervals, no significance tests are reported. Figures 2 and 3 are unreadable in the provided text, making it impossible to verify the claimed differences such as 'Bernoulli strong, adversarial least effective.' Please report numeric Hits@10 values with standard deviations across several seeds (e.g., 5-10 runs), provide the full results in a table, and regenerate the figures with larger fonts and higher resolution or as vector graphics.
- [§5.2] The adversarial sampler is evaluated only with RESCAL as the auxiliary model, and RESCAL is also one of the baseline models and one of the worst performers. The paper's own explanation that 'RESCAL was one of the lowest performing models, which significantly affected the performance of the adversarial sampler' shows that the experiment conflates the quality of the adversarial sampler with the quality of its auxiliary model. To make any claim about adversarial sampling itself, the authors should evaluate with at least one stronger auxiliary model (e.g., TransE or ComplEx) or explicitly restrict their conclusions to the specific auxiliary model used.
minor comments (6)
- [§3.1] The abstract method names appear as blank placeholders (empty boxes in the PDF). Please replace them with the actual method names (e.g., `_prepare`, `_create_negative_pool`). Also, the number of new samplers is inconsistent: §3.1 lists three ('Corrupt', 'Typed', 'Relational'), while §6 says five; clarify which five are meant and note that Nearest Neighbor and Adversarial are also implemented.
- [§1] Typo: 'extesion' should be 'extension'.
- [§3.4] Typo: 'statical analysis' should be 'statistical analysis'.
- [§4] The repository URL and version/commit identifier are missing from the manuscript; please include them for reproducibility.
- [§5.1 and §5.2] Link prediction experiments are reported only for FB15K and WN18, while YAGO4-20 and DBpedia50 are used only for pool statistics. Please either report link prediction results for the latter two datasets or explicitly state that the proof-of-concept scope is limited to two datasets.
- [Table 3] The header percentages are garbled in the provided text (e.g., '� ���'); please ensure the thresholds are clearly labeled.
Circularity Check
No circularity: the flat performance curves are an explicitly acknowledged consequence of the random-fallback supplementation rule, and no claimed result reduces to the paper's own inputs or to a self-citation chain.
full rationale
The paper is primarily a software-engineering contribution with an empirical proof-of-concept, and I find no circular step in the sense required by the review protocol. The only self-citation ([4], Barile et al.) supplies a subset of YAGO4 used as an input dataset; this is external, reproducible data, not a theorem or fitted value invoked to force a conclusion, so it is not load-bearing circularity. The central empirical finding that Hits@10 is nearly flat as the number of negatives per positive grows is explicitly explained by the authors themselves in Sect. 5.2 as a consequence of the supplementation rule described in Sect. 4.1: when a sampler's negative pool is smaller than the requested N, random entities fill the remainder, so at high N the sampling converges to random corruption. This is an arithmetic implication of the pool statistics in Table 3 (e.g., Relational sampling on WN18 has an average pool of 2, so at N=100 roughly 98% of negatives are random), not a prediction fitted to the outcome or an equation that equals its own input by construction. The paper's honesty about this limitation means it is a threat to the external validity of the empirical claim, but not circularity. Similarly, using RESCAL as both an evaluated model and the auxiliary model for adversarial sampling is an evaluation confound that the authors flag in Sect. 5.2; it does not make the derivation circular because the auxiliary model's outputs are not the paper's target result. Hyperparameters are tuned, and no uniqueness theorem or ansatz is imported from the authors' prior work to forbid alternatives. I therefore find no step where a quoted equation, parameter, or definition reduces to the claimed result by construction, and the appropriate verdict is no significant circularity.
Assumptions & free parameters
free parameters (8)
- embedding dimension =
100
- number of epochs =
100
- batch size =
500
- regularization weight lambda =
tuned in [1e-5, 1e-2]
- learning rate =
tuned in [1e-6, 1e-2]
- margin gamma =
chosen from {1,2,5,10}
- negatives per positive =
{1,2,5,20,50,100}
- auxiliary model for dynamic samplers =
RESCAL
assumptions (6)
- domain assumption Local closed-world assumption: any triple absent from the graph is treated as negative.
- domain assumption Relational sampling assumes each (head, tail) entity pair appears with at most one relation.
- domain assumption Typed sampling assumes reliable semantic type metadata (domain/range or class membership) is available and correctly aligned with entity IDs.
- domain assumption Auxiliary embeddings (RESCAL) learned without the target sampler are good guides for selecting informative hard negatives.
- ad hoc to paper Twenty hyperparameter trials with a four-hour cap per configuration reach sufficiently fair settings for comparing samplers.
- standard math Filtered Hit@10 is an adequate evaluation metric for link prediction.
Cite this review
Pith. "Pith review of Enhancing PyKEEN with Multiple Negative Sampling Solutions for Knowledge Graph Embedding Models." pith.science (2026). https://pith.science/paper/VEKZ2T5Z
@misc{pith2026250805587,
author = {Pith},
title = {Pith review of: Enhancing PyKEEN with Multiple Negative Sampling Solutions for Knowledge Graph Embedding Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/VEKZ2T5Z}},
note = {Machine review of arXiv:2508.05587}
}
read the original abstract
Embedding methods have become popular due to their scalability on link prediction and/or triple classification tasks on Knowledge Graphs. Embedding models are trained relying on both positive and negative samples of triples. However, in the absence of negative assertions, these must be usually artificially generated using various negative sampling strategies, ranging from random corruption to more sophisticated techniques which have an impact on the overall performance. Most of the popular libraries for knowledge graph embedding, support only basic such strategies and lack advanced solutions. To address this gap, we deliver an extension for the popular KGE framework PyKEEN that integrates a suite of several advanced negative samplers (including both static and dynamic corruption strategies), within a consistent modular architecture, to generate meaningful negative samples, while remaining compatible with existing PyKEEN -based workflows and pipelines. The developed extension not only enhancesPyKEEN itself but also allows for easier and comprehensive development of embedding methods and/or for their customization. As a proof of concept, we present a comprehensive empirical study of the developed extensions and their impact on the performance (link prediction tasks) of different embedding methods, which also provides useful insights for the design of more effective strategies
Reference graph
Works this paper leans on
-
[1]
Advances in Neural Information Processing Systems 33, 9649–9661 (2020)
Abboud, R., Ceylan, I., Lukasiewicz, T., Salvatori, T.: Boxe: A box embedding model for knowledge base completion. Advances in Neural Information Processing Systems 33, 9649–9661 (2020)
work page 2020
-
[2]
Alam, M.M., Jabeen, H., Ali, M., Mohiuddin, K., Lehmann, J.: Affinity dependent negative sampling for knowledge graph embeddings. (2020)
work page 2020
-
[3]
Journal of Machine Learning Research 22(82), 1–6 (2021), http://jmlr.org/papers/v22/20-825.html
Ali, M., Berrendorf, M., Hoyt, C.T., Vermue, L., Sharifzadeh, S., Tresp, V., Lehmann, J.: PyKEEN 1.0: A Python Library for Training and Evaluating Knowl- edge Graph Embeddings. Journal of Machine Learning Research 22(82), 1–6 (2021), http://jmlr.org/papers/v22/20-825.html
2021
-
[4]
In: European Semantic Web Conference
Barile, R., d’Amato, C., Fanizzi, N.: Explanation of link predictions on knowledge graphs via levelwise filtering and graph summarization. In: European Semantic Web Conference. pp. 180–198. Springer (2024)
work page 2024
-
[5]
In: Proceedings of the 2008 ACM SIGMOD international conference on Management of data
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collabo- ratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD international conference on Management of data. pp. 1247–1250 (2008)
2008
-
[6]
Advances in neural information processing systems 26 (2013)
Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. Advances in neural information processing systems 26 (2013)
work page 2013
-
[7]
In: International Workshop on Knowledge Graph: Mining Knowledge Graph for Deep Insights (Aug 2020)
Boschin, A.: Torchkge: Knowledge graph embedding in python and pytorch. In: International Workshop on Knowledge Graph: Mining Knowledge Graph for Deep Insights (Aug 2020)
work page 2020
-
[8]
Broscheit, S., Ruffinelli, D., Kochsiek, A., Betz, P., Gemulla, R.: LibKGE - A knowledge graph embedding library for reproducible research. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Sys- tem Demonstrations. pp. 165–174 (2020), https://www.aclweb.org/anthology/ 2020.emnlp-demos.22
work page 2020
Show all 41 references
-
[9]
In: Walker, M., Ji, H., Stent, A
Cai, L., Wang, W.Y.: KBGAN: Adversarial learning for knowledge graph embed- dings. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume ...
2018 doi
-
[10]
In: International Semantic Web Conference
Chen, X., Zhang, W., Yao, Z., Chen, M., Tang, S.: Negative sampling with adaptive denoising mixup for knowledge graph embedding. In: International Semantic Web Conference. pp. 253–270. Springer (2023)
2023
-
[11]
https://doi.org/10.5281/zenodo.2595043,https: //doi.org/10.5281/zenodo.2595043
Costabello, L., Bernardi, A., Janik, A., Creo, A., Pai, S., Van, C.L., McGrath, R., McCarthy, N., Tabacof, P.: AmpliGraph: a Library for Representation Learning on Knowledge Graphs (Mar 2019). https://doi.org/10.5281/zenodo.2595043,https: //doi.org/10.5281/zenodo.2595043
2019 doi
-
[12]
arXiv preprint arXiv:1908.06178 (2019)
Dash, S., Gliozzo, A.: Distributional negative sampling for knowledge base com- pletion. arXiv preprint arXiv:1908.06178 (2019)
1908 arXiv
-
[13]
Association for Computa- tional Linguistics (ACL) (2020)
Gashteovski, K., Gemulla, R., Kotnis, B., Hertling, S., Meilicke, C.: On aligning openie extractions with knowledge bases: A case study. Association for Computa- tional Linguistics (ACL) (2020)
2020
-
[14]
In: Proceedings of EMNLP (2018) Enhancing PyKEEN with Multiple Negative Sampling Solutions 17
Han, X., Cao, S., Xin, L., Lin, Y., Liu, Z., Sun, M., Li, J.: Openke: An open toolkit for knowledge embedding. In: Proceedings of EMNLP (2018) Enhancing PyKEEN with Multiple Negative Sampling Solutions 17
2018
-
[15]
arXiv preprint arXiv:2210.06242 (2022)
Je, S.H.: Entity aware negative sampling with auxiliary loss of false negative pre- diction for knowledge graph embedding. arXiv preprint arXiv:2210.06242 (2022)
2022 arXiv
-
[16]
Ji, G., He, S., Xu, L., Liu, K., Zhao, J.: Knowledge graph embedding via dynamic mapping matrix. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Lon...
2015
-
[17]
Advances in neural information processing systems31 (2018)
Kazemi,S.M.,Poole,D.:Simpleembeddingforlinkpredictioninknowledgegraphs. Advances in neural information processing systems31 (2018)
2018
-
[18]
arXiv preprint arXiv:1708.06816 (2017)
Kotnis, B., Nastase, V.: Analysis of the impact of negative sampling on link pre- diction in knowledge graphs. arXiv preprint arXiv:1708.06816 (2017)
2017 arXiv
-
[19]
In: The Semantic Web-ISWC 2015: 14th International Semantic Web Conference, Bethlehem, PA, USA, October 11-15, 2015, Proceedings, Part I
Krompaß, D., Baier, S., Tresp, V.: Type-constrained representation learning in knowledge graphs. In: The Semantic Web-ISWC 2015: 14th International Semantic Web Conference, Bethlehem, PA, USA, October 11-15, 2015, Proceedings, Part I
2015
-
[20]
pp. 640–655. Springer (2015)
2015
-
[21]
Semantic web6(2), 167–195 (2015)
Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P.N., Hellmann, S., Morsey, M., Van Kleef, P., Auer, S., et al.: Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web6(2), 167–195 (2015)
2015
-
[22]
In: 2023 China Automation Congress (CAC)
Li, D., Qu, H., Wang, J.: A survey on knowledge graph-based recommender sys- tems. In: 2023 China Automation Congress (CAC). pp. 2925–2930. IEEE (2023)
2023
-
[23]
In: Proceedings of the AAAI conference on artificial intelligence
Lin, Y., Liu, Z., Sun, M., Liu, Y., Zhu, X.: Learning entity and relation embed- dings for knowledge graph completion. In: Proceedings of the AAAI conference on artificial intelligence. vol. 29 (2015)
2015
-
[24]
arXiv preprint arXiv:2402.19195 (2024)
Madushanka, T., Ichise, R.: Negative sampling in knowledge graph representation learning: A review. arXiv preprint arXiv:2402.19195 (2024)
2024 arXiv
-
[25]
Advances in neural information processing systems26 (2013)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed repre- sentations of words and phrases and their compositionality. Advances in neural information processing systems26 (2013)
2013
-
[26]
Communications of the ACM 38(11), 39–41 (1995)
Miller, G.A.: Wordnet: a lexical database for english. Communications of the ACM 38(11), 39–41 (1995)
1995
-
[27]
Proceedings of the IEEE 104(1), 11–33 (2016)
Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E.: A review of relational machine learning for knowledge graphs. Proceedings of the IEEE 104(1), 11–33 (2016). https://doi.org/10.1109/JPROC.2015.2483592
2016
-
[28]
In: Proceedings of the AAAI conference on artificial intelligence
Nickel, M., Rosasco, L., Poggio, T.: Holographic embeddings of knowledge graphs. In: Proceedings of the AAAI conference on artificial intelligence. vol. 30 (2016)
2016
-
[29]
In: Icml
Nickel, M., Tresp, V., Kriegel, H.P., et al.: A three-way model for collective learning on multi-relational data. In: Icml. vol. 11, pp. 3104482–3104584 (2011)
2011
-
[30]
In: The Semantic Web: 17th International Conference, ESWC 2020, Herak- lion, Crete, Greece, May 31–June 4, 2020, Proceedings 17
Pellissier Tanon, T., Weikum, G., Suchanek, F.: Yago 4: A reason-able knowledge base. In: The Semantic Web: 17th International Conference, ESWC 2020, Herak- lion, Crete, Greece, May 31–June 4, 2020, Proceedings 17. pp. 583–596. Springer (2020)
2020
-
[31]
In: European Semantic Web Conference
Senaratne,A.,Omran,P.G.,Christen,P.,Williams,G.:Tric:Atriplescorrupterfor knowledge graphs. In: European Semantic Web Conference. pp. 117–122. Springer (2023)
2023
-
[32]
Advances in neural information processing systems 26 (2013)
Socher, R., Chen, D., Manning, C.D., Ng, A.: Reasoning with neural tensor net- works for knowledge base completion. Advances in neural information processing systems 26 (2013)
2013
-
[33]
arXiv preprint arXiv:1902.10197 (2019) 18 Claudia d’Amato, Ivan Diliso, Nicola Fanizzi, and Zafar Saeed
Sun, Z., Deng, Z.H., Nie, J.Y., Tang, J.: Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197 (2019) 18 Claudia d’Amato, Ivan Diliso, Nicola Fanizzi, and Zafar Saeed
1902 arXiv
-
[34]
In: International conference on machine learning
Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., Bouchard, G.: Complex embed- dings for simple link prediction. In: International conference on machine learning. pp. 2071–2080. PMLR (2016)
-
[35]
Information Sciences606, 853– 863 (2022)
Wang, C., Zhu, Z., Meng, P., Qiu, Y.: Leveraging network structure for efficient dynamic negative sampling in network embedding. Information Sciences606, 853– 863 (2022)
2022
-
[36]
In: Proceedings of the AAAI conference on artificial intelligence
Wang,Z.,Zhang,J.,Feng,J.,Chen,Z.:Knowledgegraphembeddingbytranslating on hyperplanes. In: Proceedings of the AAAI conference on artificial intelligence. vol. 28 (2014)
2014
-
[37]
arXiv preprint arXiv:1412.6575 (2014)
Yang, B., Yih, W.t., He, X., Gao, J., Deng, L.: Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575 (2014)
2014 arXiv
-
[38]
Advances in neural information processing systems32 (2019)
Zhang, S., Tay, Y., Yao, L., Liu, Q.: Quaternion knowledge graph embeddings. Advances in neural information processing systems32 (2019)
2019
-
[39]
In: 2019 IEEE 35th International Con- ference on Data Engineering (ICDE)
Zhang, Y., Yao, Q., Shao, Y., Chen, L.: Nscaching: simple and efficient negative sampling for knowledge graph embedding. In: 2019 IEEE 35th International Con- ference on Data Engineering (ICDE). pp. 614–625. IEEE (2019)
2019
-
[40]
In: Proceed- ings of the 43rd International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval
Zheng, D., Song, X., Ma, C., Tan, Z., Ye, Z., Dong, J., Xiong, H., Zhang, Z., Karypis, G.: Dgl-ke: Training knowledge graph embeddings at scale. In: Proceed- ings of the 43rd International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval. p. 739–748....
2020
-
[41]
In: The World Wide Web Conference
Zhu, Z., Xu, S., Qu, M., Tang, J.: Graphvite: A high-performance cpu-gpu hybrid system for node embedding. In: The World Wide Web Conference. pp. 2494–2504. ACM (2019)
2019
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.