{"id":"1be4749f-3321-4a10-bf54-80b5a52599c6","arxiv_id":"2505.13098","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-KG-Bench 3.0 is an open, extensible benchmark framework with automatic scoring that compares more than 30 LLMs on RDF and SPARQL knowledge graph tasks.","lead":"Researchers released LLM-KG-Bench 3.0, an open benchmark that automatically tests how well large language models handle knowledge graph formats such as RDF and SPARQL. It includes evaluation results from more than 30 chat models and a public dataset of prompts and answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline capability comparisons rest on contentF1 and a hand-weighted combined score that are never validated against human expert judgment; any systematic evaluator bias would propagate into every model card and format comparison.","rationale":"The paper's contribution is an automated benchmark plus a published dataset for comparing LLMs on RDF/SPARQL tasks. The central claim of usefulness depends on the automated scores being faithful to knowledge graph engineering quality. That condition is least secure in Section 3.4, where contentF1 and the combined score are defined but never validated against human judgment, and it is also the weakest assumption identified by the reader. The concern is not that the scores disagree with external consensus; it is that the paper provides no evidence for the correctness of its own evaluator, and any evaluator bias would flow directly into every reported model comparison, model card, and format-preference test in Table 3. Because this is exactly the concern the reader already flagged, and because the reader's CONDITIONAL verdict already accounts for it, my pass does not change the verdict. The open code and released dataset are genuine independent support that the framework operates and produces artifacts, but artifact existence is weaker than metric validity; a human-expert spot-check is the direct and feasible test that would settle the concern.","tokens_in":13553,"tokens_out":4542,"duration_ms":47679,"concrete_test":"Select a stratified random sample of 100 task iterations from the published dataset, covering all task classes and a spread of model sizes. Have two Semantic Web experts independently score each LLM answer for correctness using a task-specific rubric, blind to the automated scores; then compute Cohen's kappa between the experts and the Spearman rank correlation between the mean expert score and the automated contentF1/combined score per task and overall. If per-task rank correlation is below about 0.8, or if experts disagree with automated scores on a substantial fraction of cases, the benchmark's comparisons are not yet supported as capability measures. A supplementary check is to recompute contentF1 with an independent RDF canonicalization (for example, rdflib graph isomorphism) to determine whether normalized-triple equality is the source of disagreement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM-KG-Bench 3.0 provides automated evaluation as a reliable basis for comparing LLM capabilities in RDF and SPARQL. The load-bearing assumption is that the automated scoring functions—contentF1 on normalized triples and the combined score 0.1*strSimilarity + 0.2*parsableSyntax + 0.7*contentF1 defined in Section 3.4—measure what a human expert would judge as correct knowledge graph engineering. The paper reports no validation against human judgment, no inter-annotator agreement study, no qualitative error analysis, and no independent reimplementation of the normalized-triple F1. Because every model card, capability compass, and the TTL/JSON-LD t-tests in Table 3 are derived from these scores, a systematic evaluator bias (for example, penalizing semantically equivalent reorderings, blank-node labelling, or alternative valid SPARQL formulations that the normalizer does not recognize) would propagate to all reported model comparisons. This is an internal-correctness risk rather than an external-consensus objection, since the paper's own contribution is the automated evaluator and its validity is left unexamined. The open code and released dataset are real evidence that the framework runs and produces artifacts, but they do not by themselves establish that the scores faithfully capture KG engineering capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes LLM-KG-Bench 3.0, an open-source framework for automated evaluation of LLM capabilities on knowledge graph engineering tasks, including RDF serialization (Turtle, JSON-LD, N-Triples, RDF/XML) and SPARQL syntax and semantics. It introduces a revised Task API, an RDF-syntax repair task with dialogue-based correction cycles, encrypted task data to avoid test-set leakage, a vLLM connector, and capability compass visualizations. The authors apply the framework to evaluate more than 30 open and proprietary LLMs and publish the resulting prompts, answers, and evaluation scores, along with exemplary model cards and pairwise t-tests comparing Turtle versus JSON-LD performance.","tokens_in":13884,"tokens_out":5096,"duration_ms":48231,"significance":"If the automated scores are a valid proxy for human-judged knowledge graph engineering capability, this is a valuable community resource: it is the first benchmark framework that combines automatic evaluation with a broad set of KGE topics and many LLMs, and the released code and data (with DOIs) are concrete and reproducible. The framework's extensibility, the public dataset of model interactions, and the machine-checked evaluation pipeline are strong assets. However, the central validity claim for the automated evaluator is currently unsubstantiated, so the significance is conditional on the missing validation evidence.","major_comments":[{"comment":"The automated evaluation scores defined in Section 3.4 (parsableSyntax, contentF1 computed on normalized triples, strSimilarity, and the combined score 0.1*strSimilarity + 0.2*parsableSyntax + 0.7*contentF1) are used to produce the capability compass plots and the Turtle-versus-JSON-LD t-test results in Table 3, but the paper provides no validation that these scores agree with human expert judgment. A systematic evaluator bias, for example a normalizer that penalizes semantically equivalent triple reorderings or alternative blank-node labels, would propagate into every model comparison derived from these scores. Please add a human-judgment validation study, an inter-annotator agreement analysis, or at least a qualitative error analysis of the automated evaluation, and discuss the sensitivity of the hand-weighted combined score.","section":"3.4"},{"comment":"Table 3 reports two-sided t-test preferences between Turtle and JSON-LD, but the paper does not state which score was used as the dependent variable (e.g., contentF1, combined, or per-task scores), whether the tests were paired or independent, or how multiple comparisons across 33 models and six tasks were corrected. Without this information, the format-preference results are not reproducible from the description in the paper. Please specify the exact test procedure, the score, the sample sizes per model and task, and the multiple-testing correction.","section":"4.2, Table 3"},{"comment":"The selection of 20 iterations for proprietary models and 50 iterations for open models is presented as a trade-off, but the paper does not show how this asymmetry affects the statistical power of the comparisons or the reported preference tests in Table 3. Since the dataset is itself a central contribution, the paper should include a comparison of effect sizes or confidence intervals for at least the headline models, to demonstrate that the iteration counts are sufficient for the claims made.","section":"4.1"}],"minor_comments":[{"comment":"The title on the arXiv page reads \"SemanticTechnology\" without a space; it should be \"Semantic Technology\".","section":"Title"},{"comment":"In the combined score formula, \"contentF 1\" contains a stray space; it should read \"contentF1\".","section":"3.4"},{"comment":"The capability compass plots are too small to read the dimension labels clearly; please enlarge them or provide a tabular summary of the aggregated scores.","section":"4.2, Fig. 4"},{"comment":"The row for LLM-KG-Bench 3 reads \"many several automatic\"; this should be clarified, for example by giving the number of models and tasks covered.","section":"2, Table 1"},{"comment":"The description of RdfSyntaxFixList states that there are five task case entries for each serialization format; please also state the total number of task cases per format and how they are selected across iterations.","section":"3.4"},{"comment":"The date notation in Table 3 (e.g., \"2024/01\") is not explained; clarifying that it denotes the model snapshot would help reproducibility.","section":"4.1, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the ESWC Resources Track well. The main concern is the lack of validation of the automated evaluator; I recommend asking the authors to provide a validation study or to substantially soften the claims about reliable capability comparison. The t-test methodology in Table 3 must also be clarified. The open code and published dataset are genuine strengths."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid resource paper, not a paradigm shift, and it mostly earns its keep. The genuinely new things are the RdfSyntaxFixList task spanning Turtle/JSON-LD/N-Triples, the vLLM connector, encrypted task data to reduce leakage, and a published dataset of prompts, answers, and scores for more than 30 open and proprietary LLMs. The code and data are public and DOI'd, which makes the central claim — that the framework runs and produces an inspectable evaluation corpus — credible and independently checkable. That matters; a lot of benchmark papers are all promise and no artifact.\n\nThe paper also does well in framing itself as a living toolkit rather than a one-shot leaderboard. The updated task API and the re-evaluation mode are pragmatic features that will make it easier for other groups to add tasks. The capability compass plots are a nice way to summarize model strengths across task categories.\n\nThe soft spot is exactly what the reader and the stress-test emphasize: contentF1 on normalized triples and the hand-weighted combined score (0.1 strSimilarity + 0.2 parsableSyntax + 0.7 contentF1) are taken as proxies for knowledge graph engineering capability, but there is no validation against human experts, no inter-annotator agreement study, and no error analysis showing that the normalizer handles semantically equivalent reorderings or blank-node labels correctly. That is a real limitation, not a quibble, because every model comparison and every TTL-vs-JSON-LD t-test in Table 3 inherits whatever systematic bias the evaluator has. The open dataset mitigates this somewhat — I can re-score the raw answers myself — but the paper does not do that homework, and the absence of confidence intervals for the headline scores makes it harder to know how much of the difference is noise, even with 20 vs 50 iterations.\n\nI disagree with one point in the stress-test note: it says the validity is left unexamined, but the paper does clearly define its metrics and does publish the raw data, so it isn't hiding anything. The issue is that the authors stop short of demonstrating that the scores mean what they claim. That is a gap, but not a fatal one for a resource paper.\n\nWho should read this: anyone in the Semantic Web community who wants a reusable harness for LLM evaluation on RDF/SPARQL tasks, and anyone designing automated benchmark scoring in related domains. It deserves a serious referee, and already got one at ESWC; the right outcome is an accept with a request for a validation appendix.\n\nMy verdict: worth citing, worth having in the literature, but I would phrase any claims about model capability as \"as measured by LLM-KG-Bench\" until the metrics are independently validated.","headline":"A genuinely useful open benchmark for LLM+KGE with a clear soft spot: the automated scoring metrics are never checked against human judgment, so the headline comparisons rest on unvalidated proxies.","tokens_in":14335,"tokens_out":1501,"would_cite":true,"duration_ms":17254,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A public benchmark automatically scores 30+ LLMs on RDF and SPARQL skills.","keywords":["LLM evaluation","RDF","SPARQL","knowledge graph engineering","semantic technologies","benchmark framework","Turtle","JSON-LD"],"falsifier":"Have human semantic-web experts score a random sample of the published model answers on the same tasks, then compare their scores to the framework's automated scores; if the two disagree systematically, the reported rankings would not survive.","tokens_in":13319,"feed_emoji":"🧭","tokens_out":3782,"duration_ms":36548,"temperature":0.7,"pith_summary":"The paper aims to establish that LLM-KG-Bench 3.0 is a working, extensible benchmark for automatically scoring how well large language models handle knowledge-graph engineering tasks: generating and repairing RDF in multiple serializations, answering questions over small graphs, and writing SPARQL queries. If that claim holds, researchers can compare dozens of LLMs on semantic-technology skills without manually reading answers, and can track capability changes as models are updated. The paper backs the claim with a released framework, a dataset of prompts, answers, and scores for more than 30 open and proprietary models, and exemplary model cards. A sympathetic reader would treat the contribution as a public measurement resource and a method for further knowledge-graph engineering benchmarking.","feed_headline":"Compass ranks 30+ LLMs on RDF and SPARQL skills","feed_subtitle":"Automated tasks score models on syntax, analytics, SPARQL semantics, and brevity; open code and data included.","key_machinery":"The prompt-answer-evaluate loop, orchestrated by the task API, is the load-bearing mechanism. Each task supplies the prompt and the evaluator; the model connector returns an answer; evaluation determines whether another correction round starts. For syntax-repair tasks, scoring is composite: parsableSyntax with weight 0.2, contentF1 on normalized triples with weight 0.7, string similarity with weight 0.1, plus a brevity score for extraneous text, with contentF1 computed only when the document parses. The capability compass spider plot aggregates these scores into per-model cards.","core_discovery":"The paper claims that the LLM-KG-Bench framework, in Version 3.0, provides an extensible, automated evaluation pipeline for LLM capabilities in semantic technologies. Applying this pipeline to more than 30 open and proprietary LLMs yields a public dataset of prompts, answers, and evaluation scores, and aggregate scores can be visualized as capability compasses across dimensions such as RDF syntax, RDF analytics, SPARQL syntax, SPARQL semantics, and brevity. The framework's design enables dialogue-based correction cycles: when a generated RDF or SPARQL answer fails parsing or content checks, the LLM is asked to revise it for up to three rounds, and final scores reflect the corrected answer.","pith_inferences":["The weighted combined score places 70% of its weight on contentF1, so a model that produces correct triples in an equivalent but differently ordered or named form may be penalized unless normalization captures that equivalence.","The same task API could be extended to other semantic-technology artifacts, such as SHACL shapes, OWL axioms, or R2RML mappings, though the paper does not propose those tasks.","The observed format-preference differences between models could inform serialization-specific prompting strategies, but the paper only reports the statistical differences, not how to exploit them.","Because the automated scores are not validated against expert human judgment, the rankings should be read as framework-internal scores until agreement with human assessment is demonstrated."],"forward_implications":["New LLMs can be plugged in through existing connectors and compared on the same tasks without manual answer checking.","The published dataset allows others to rerun evaluations and re-evaluate stored answers with updated scoring code, without new LLM calls.","Serialization preferences between Turtle and JSON-LD can be detected statistically for individual models using paired t-tests.","Encrypted task data reduces the risk that benchmark prompts leak into training corpora.","Capability compass plots provide compact, comparable model cards for RDF and SPARQL proficiency."],"supporting_citations":[{"why":"Supplies the original framework design and benchmark architecture that Version 3 extends.","marker":"[20]"},{"why":"Introduced the Turtle capability tasks and early open and proprietary model results that Version 1.1 built on.","marker":"[8]"},{"why":"Added SPARQL tasks and evaluated proprietary SPARQL capabilities, forming the basis for Version 2.0.","marker":"[19]"},{"why":"Tracked 2023 capability changes across LLM versions and motivates the re-evaluation mode.","marker":"[9]"},{"why":"Provides manual ChatGPT knowledge-graph engineering experiment results that the automated benchmark builds on and automates.","marker":"[21]"},{"why":"Supplies an automated text-to-knowledge-graph benchmark used as a contrast for the framework's scope.","marker":"[22]"},{"why":"Guides selection of open LLMs through average benchmark scores.","marker":"[7]"},{"why":"Guides selection of proprietary model families through crowd-preference rankings.","marker":"[5]"},{"why":"Provides the high-throughput serving engine used as a connector for running open models.","marker":"[16]"},{"why":"Offers a text-to-SPARQL generation approach that informs the SPARQL task design.","marker":"[15]"}],"fun_headline_variants":["Auto-benchmark ranks 30+ LLMs on RDF and SPARQL","LLM-KG-Bench 3.0: 30+ models scored on semantic skills","Which LLM masters RDF and SPARQL? Auto-eval says","RDF and SPARQL: new benchmark auto-grades 30+ LLMs","LLM-KG-Bench 3.0: automated compass for KG capabilities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automated scoring is assumed to reflect genuine knowledge-graph engineering ability, yet the paper never checks those scores against human expert judgment; if an automated evaluator is biased, every model comparison inherits the bias.","fun_headline_variants_meta":{"raw":{"variants":["Auto-benchmark ranks 30+ LLMs on RDF and SPARQL","LLM-KG-Bench 3.0: 30+ models scored on semantic skills","Which LLM masters RDF and SPARQL? Auto-eval says","RDF and SPARQL: new benchmark auto-grades 30+ LLMs","LLM-KG-Bench 3.0: automated compass for KG capabilities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001236,"raw_usage":{"total_tokens":5071,"prompt_tokens":935,"completion_tokens":4136,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":4026}},"tokens_in":551,"tokens_out":4136,"duration_ms":28775,"temperature":1.0,"reasoning_tokens":4026,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:19:00.419959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human semantic-web experts score a random sample of the published model answers on the same tasks, then compare their scores to the framework's automated scores; if the two disagree systematically, the reported rankings would not survive.","supporting_citations":[{"cited_title":"In: Vakaj, E., Iranmanesh, S., Stamartina, R., Mihindukula- sooriya, N., Tiwari, S., Ortiz-Rodríguez, F., Mcgranaghan, R","cited_arxiv_id":null,"evidence_quote":"Supplies the original framework design and benchmark architecture that Version 3 extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduced the Turtle capability tasks and early open and proprietary model results that Version 1.1 built on."},{"cited_title":"In: Keshan, N., Neumaier, S., Gentile, A.L., Vahdati, S","cited_arxiv_id":null,"evidence_quote":"Provides manual ChatGPT knowledge-graph engineering experiment results that the automated benchmark builds on and automates."},{"cited_title":"https://huggingface.co/spaces/open-llm-leaderboard/open_llm_ leaderboard (2024)","cited_arxiv_id":null,"evidence_quote":"Guides selection of open LLMs through average benchmark scores."},{"cited_title":"In: Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., Berkenkamp, F","cited_arxiv_id":null,"evidence_quote":"Guides selection of proprietary model families through crowd-preference rankings."},{"cited_title":"In: International Conference on Semantic Systems","cited_arxiv_id":null,"evidence_quote":"Provides the high-throughput serving engine used as a connector for running open models."}],"review_version":1}