Protoknowledge Shapes Behaviour of LLMs in Downstream Tasks: Memorization and Generalization with Knowledge Graphs

Andrea Zugarini; Fabio Massimo Zanzotto; Federico Ranaldi; Leonardo Ranaldi

REVIEW 4 major objections 5 minor 10 references

Protoknowledge Shapes Behaviour of LLMs in Downstream Tasks: Memorization and Generalization with Knowledge Graphs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a model's success at recalling knowledge-graph triples—topological protoknowledge—determines whether it can produce correct Text-to-SPARQL queries when no URI hints are given, and that this agreement exposes…

desk verdict Useful probe framework, but the Section 5.1 Agreement/Disagreement contrast is a base-rate artifact — the central 'shapes behaviour' claim needs a marginals-controlled dependence measure. read the letter →

arxiv 2505.15501 v1 pith:3BBTSSM5 submitted 2025-05-21 cs.CL cs.AI

Federico Ranaldi , Andrea Zugarini , Leonardo Ranaldi , Fabio Massimo Zanzotto This is my paper · ORCID

classification cs.CLcs.AI

keywords protoknowledgeknowledgegraphsLLMmemorizationgeneralizationText-to-SPARQLActivationTaskssemanticbiasdatacontamination

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a concept it calls protoknowledge: the structured knowledge a language model stores about a knowledge graph during pretraining, at three levels—lexical (label-to-URI mappings), hierarchical (class-subclass relations), and topological (which triples actually hold). It tries to establish that these levels can be measured separately with simple recall tasks and that the measured level predicts downstream behavior: in Text-to-SPARQL, models that correctly complete the relevant triples are the ones that generate correct queries when no URI hints are supplied. The paper's headline evidence is that, in the No-URI setting, agreement between correct triple completion and correct query generation consistently outweighs disagreement across models and graphs. If the claim holds, it gives researchers a black-box way to infer whether a closed model is recalling memorized semantic content—semantic-level data contamination—rather than reasoning from context. A careful reader should care because contamination audits for models with undisclosed training data are otherwise hard to perform.

What carries the argument

The machinery is a per-example correlation between two measurements. The first is a set of Knowledge Activation Tasks (KATs), each designed to isolate one protoknowledge form: URI recognition (lexical), direct and inverse subsumption (hierarchical), and SV?/?VO triple completion (topological). The second is the Speculative Protoknowledge for SPARQL (SPS) score, defined as $SPS = |T_{\text{predicted}} \cap T_{Q}| / |T_{Q}|$, which measures how many entity-property pairs from a gold query the model can reconstruct as valid triples, verified by SPARQL ASK queries. These two measurements are then combined in a Positive Agreement/Disagreement framework that classifies each query by whether correct query generation and correct protoknowledge activation coincide, which is what lets the paper attribute query success to memorized structure.

What would settle it

Take a structurally identical set of questions to QALD-9 but replace all entity and relation URIs with random, semantically uninformative identifiers, and run the No-URI condition. If a model with near-zero SPS still produces correct queries at non-trivial rates, or if Agreement no longer exceeds Disagreement, the claimed dependence on memorized protoknowledge would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that LLMs absorb knowledge-graph content as reusable protoknowledge in three distinguishable forms, and that the activation of each form can be measured by Knowledge Activation Tasks (KATs): label-to-URI recognition for lexical, direct and inverse subsumption for hierarchical, and subject-verb-object triple completion for topological protoknowledge. The paper's central result is that these measurements align with Text-to-SPARQL behavior: in the No-URI condition, successful query generation and successful triple completion agree more often than they disagree, and Positive Agreement—correct queries accompanied by correct protoknowledge activation—often reaches high ratios, including 100% for GPT-3.5 Turbo. The same alignment is present, though weaker, for lexical protoknowledge in the No-Label condition and for hierarchical protoknowledge on DBpedia. The paper presents this query-by-query agreement as a practical tool for exposing semantic-level data contamination in closed-pretraining models.

Load-bearing premise

The load-bearing premise is that a correct SPARQL query implies the model actually recalled the relevant knowledge-graph triples (and, for topological protoknowledge, that correct recall is required for a correct query); if query success can come from surface pattern-matching or from copying URIs in the prompt, the agreement statistics do not measure protoknowledge.

Editorial extensions

If this is right

In the No-URI condition, a model's Text-to-SPARQL performance on QALD-9 can be anticipated from its SPS triple-completion scores: models that fail to recall a query's triples will generally fail to produce the query.
In the Original condition, where URIs and labels are supplied, correct query generation no longer requires protoknowledge, so benchmarks that provide full context underestimate the role of memorized structure.
Lexical protoknowledge (label-to-URI mapping) is what lets models keep working in the No-Label condition, especially on Wikidata's non-human-readable identifiers.
The per-example Positive Agreement ratio offers a black-box profile of semantic-level data contamination for closed-pretraining models, complementing verbatim-memorization checks.
Because protoknowledge is stronger for popular items, performance on frequent entities and properties should not be extrapolated to rare or out-of-distribution content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same query-by-query agreement framework could be adapted to other structured tasks, such as Text-to-SQL, by defining schema protoknowledge through recall tasks over tables, columns, and foreign-key triples.
If protoknowledge is frequency-driven, contamination audits should stratify Positive Agreement by entity popularity; a sharp drop on rare entities would indicate memorization of common web content rather than robust generalization.
A testable extension the paper does not run: rerun the No-URI condition with entity and relation URIs replaced by random strings; a collapse in Agreement would confirm that success depends on memorized token-level structure rather than abstract graph reasoning.
The paper's deliberate omission of the S?O triple-completion task suggests a full three-slot SPS would be an even stronger predictor, provided overly frequent properties are controlled for.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper gives a clean way to think about three levels of KG knowledge (lexical, hierarchical, topological) and a coherent set of probes. The KATs are sensible, the popularity effect is consistent across tasks, and the limitations section is honest. But the load-bearing evidence for the headline claim — that protoknowledge 'shapes' Text-to-SPARQL and can expose semantic contamination — does not hold up. The Section 5.1 / Figure 3 comparison of Agreement vs Disagreement conflates true dependence with marginal success rates. Under independence, when both query success and SPS success are low (as in No URI), Agreement dominates; the paper's own numbers (e.g., query F1 of 3.7–29.1%, SPS perfect scores mostly below 30%) already predict that. No dependence measure controlling for marginals is reported, so the plotted contrast carries no information about whether triple completion and query generation are related. The paper's Original condition, where Disagreement dominates, also contradicts the stated assumption that a correct SPARQL query implies correct topological protoknowledge. That assumption is doing real work, and the data refute it.

What's new and worth keeping: the protoknowledge taxonomy is a reasonable organizing concept, and applying it to Text-to-SPARQL contamination is new. The URI recognition, subsumption, and triple-completion tasks are a coherent probe suite. The popularity/semantic-bias result is plausible and appears across several tasks, though it is mostly descriptive.

Soft spots in proportion: small samples (2–13 examples per cell in places), no significance tests, post hoc popularity splits, and the lexical PA analysis discards below-median items without justification. Table 1 has inconsistent percentages and counts. None of these kill the framework; they just mean the quantitative claims are fragile.

Bottom line: as a methodology proposal, the paper deserves referee time. As a demonstration that protoknowledge shapes downstream behavior, it needs a proper dependence measure (e.g., compare Agreement to the independence baseline) and ideally a validation against known contamination. If the authors redo the analysis that way, the framework could become solid. Send it to review, but with a clear request to fix the base-rate problem.

Referee Report

4 major / 5 minor

Summary. The paper introduces the notion of 'protoknowledge' to describe how LLMs internalize Knowledge Graph content during pretraining and reuse it in downstream tasks. It distinguishes lexical, hierarchical, and topological forms, measures them through Knowledge Activation Tasks (URI recognition, subsumption, and triple completion), and analyzes their dependence on item popularity as evidence of semantic bias. It then examines Text-to-SPARQL performance under Original, No Label, and No URI prompting conditions and uses a Positive Agreement framework to claim that success in triple completion predicts success in query generation, particularly when URI hints are absent. The paper concludes that this framework offers a practical tool for detecting semantic-level data contamination in closed-pretraining models.

Significance. If the central claim were supported, the paper would provide a useful black-box method for linking internal KG recall to structured query generation and for probing contamination in models whose pretraining data is unavailable. The taxonomy of lexical, hierarchical, and topological protoknowledge and the KAT design are a reasonable starting point for such an analysis. However, the evidence as presented does not establish the claimed dependence: the main Agreement/Disagreement statistic is confounded by base rates, and the paper's own assumption about query success implying protoknowledge activation is contradicted by its Original-mode results. A reanalysis with margin-controlled dependence measures could make the contribution meaningful, but the current support is inadequate.

major comments (4)

[§5.1 and Figure 3] The central evidence that 'Agreement consistently exceeds Disagreement' in the No URI setting is confounded by the marginal success rates. Under independence of query success (probability p) and SPS success (probability q), P(Agreement) = pq + (1-p)(1-q) and P(Disagreement) = p(1-q) + (1-p)q. In the No URI setting, Table 4 reports F1 values of 3.67–29.09 and Table 8 reports perfect SPS values mostly below 30%; for example, p=0.1 and q=0.2 already predict 74% Agreement against 26% Disagreement. The plotted contrast therefore carries no information about whether triple completion and query generation are statistically or causally related. The paper should report a margin-controlled statistic (e.g., Cohen's kappa, a chi-square test on the 2×2 table, or a permutation baseline) and should compare the observed Agreement against the independence prediction rather than against Disagreement alone.
[§5.1, Figure 3, Original setting] The framework assumes that 'a correct SPARQL query implies successful activation of protoknowledge', and for topological protoknowledge it assumes the reverse as well. This is directly contradicted by the same figure: in the Original setting Disagreement dominates, which must include cases where the SPARQL query is correct while the corresponding triple completion is incorrect. If correct queries can be produced without successful triple recall, then query performance does not require the measured protoknowledge, and the Positive Agreement ratio is merely a conditional frequency rather than evidence of mediation. The paper must report the full 2×2 contingency table (query correct/incorrect by SPS correct/incorrect) and separate the two directional cells of Disagreement.
[§4.4 and Limitations] Equation (1) defines SPS as |Tpredicted ∩ TQ| / |TQ|, where TQ is 'all entity-property pairs relevant to Qgold' and Tpredicted comes from the SV? and ?VO tasks. The Limitations paragraph states that the S?O task is deliberately omitted. This makes the metric's coverage unclear: if TQ contains triples whose property must be inferred from a subject-object pair, those triples can never appear in Tpredicted, so SPS will be systematically depressed and the Agreement analysis will be distorted. The paper should either restrict TQ to entity-property pairs testable by SV? and ?VO or report the proportion of Qgold triples covered by these two tasks.
[§4.2, Table 1] The quantitative evidence for the popularity trend is not auditable in its current form. In the LF row for entities, the cell '3.11% (5/9)' is internally inconsistent: 5/9 is 55.6%, while 5/161 is 3.11%. Several other cells show the same mismatch between percentage and parenthetical ratio. Additionally, the hierarchical results in Tables 2 and 3 rest on supports as small as 2–5 items and are reported without any significance or confidence intervals. The semantic-bias claims need exact counts, internally consistent percentages, and at least exact binomial confidence intervals or an explicitly stated exploratory status.

minor comments (5)

[§5.1, lexical PA analysis] The lexical Positive Agreement analysis excludes items below the 50th percentile of popularity without justification, and Table 12's explanation of the threshold is unclear; the paper should state the selection rule explicitly and, if possible, also report results on the full set.
[Equation (1)] TQ and Tpredicted are not formally defined as sets, and the role of Perfect versus Soft matches in the SPS score is described only in prose; a precise set-theoretic definition would improve reproducibility.
[Figure 3] The bars do not show the number of examples behind each percentage, even though the text notes that some ratios rest on very small counts (e.g., three examples for Llama-3_70B); adding per-bar n values is essential for interpreting the figure.
[Section 5.1 and Table 6] The text states Llama-3_70B has hierarchical PA of 32/39, while Table 6 lists 32/40; these should be harmonized.
[Throughout] There are several typos and minor wording issues, including 'categorizeprotoknowledge' in the abstract, 'hypotesize' in Section 5.1, 'benchmkarks' and 'Hierachical' in the Limitations paragraph, and the nonstandard use of 'overconfidence' in Section 4.5 to mean accuracy bias rather than confidence calibration.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper correlates two independently measured quantities; the Agreement statistic is a consistency check, not an identity or a fitted prediction.

full rationale

The paper’s chain is: define protoknowledge forms, measure them with Knowledge Activation Tasks, measure Text-to-SPARQL performance, then compare the two per query. Each measurement is obtained directly from model outputs on constructed test sets, and neither quantity is fitted to the other. The SPS score in Eq. 1 is defined from gold triples and model predictions; the Text-to-SPARQL F1 is computed from generated queries against gold answers. The Positive Agreement/Agreement statistics in Section 5.1 are conjunctions and disjunctions of these two measurements, not quantities derived from one another. No load-bearing parameter is fitted to the target result. The self-citation of Ranaldi et al. (2024) motivates the contamination setting but is not used as evidence for the present experiments. A statistical concern—that Agreement versus Disagreement is sensitive to base rates, so high agreement at low success rates can arise under independence—is a validity threat to the paper’s interpretation, not circularity. It does not make the derivation equivalent to its inputs. No circular step is exhibited.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central analysis depends on three unverified assumptions: (1) pretraining corpora resemble web corpora, (2) query success implies internal triple recall, and (3) knowledge-graph triple counts approximate pretraining exposure. No independent evidence is supplied for these, and the paper's own Original-condition results weaken assumption 2.

free parameters (3)

popularity threshold for frequent/infrequent split = mean or median triple-count threshold
Used throughout Section 4 to partition items; reported accuracy trends depend on which threshold is chosen and where it is applied.
50th-percentile exclusion in lexical PA = exclude items below median popularity
Section 5.1: 'The analysis excludes items below the 50th percentile of popularity'; this post-hoc filter boosts Positive Agreement ratios.
soft match tolerance in SPS = Soft matches counted when URI is related to correct entity via a different property
Section 4.4 defines Perfect vs Soft matching; the Loose SPS score includes soft matches, changing the correlation with query success.

assumptions (3)

domain assumption Pretraining corpora of Closed-Pretraining models broadly reflect common web data
Section 3.2: 'we assume, consistent with prior observations, that their content distribution broadly reflects common web data'; this turns KG item popularity into a proxy for pretraining exposure.
ad hoc to paper A correct SPARQL query implies successful activation of the relevant protoknowledge; for topological protoknowledge the reverse also holds
Section 5.1 states this assumption and uses it to interpret Positive Agreement as evidence that protoknowledge shapes query generation; the Original-prompt results where Disagreement dominates contradict the implication.
domain assumption Triple-count popularity in DBpedia and Wikidata is a valid proxy for frequency in pretraining data
Section 4.1 uses number of triples referencing an item to define popularity and to infer 'semantic bias' from accuracy differences.

invented entities (1)

protoknowledge
purpose: A latent capability of LLMs to memorize and functionally reuse KG content from pretraining, categorized into lexical, hierarchical, and topological forms
Defined in Section 3 and measured only through the paper's own Knowledge Activation Tasks; no independent falsifiable handle outside the proposed framework.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Protoknowledge Shapes Behaviour of LLMs in Downstream Tasks: Memorization and Generalization with Knowledge Graphs." pith.science (2026). https://pith.science/paper/3BBTSSM5

@misc{pith2026250515501,
  author       = {Pith},
  title        = {Pith review of: Protoknowledge Shapes Behaviour of LLMs in Downstream Tasks: Memorization and Generalization with Knowledge Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3BBTSSM5}},
  note         = {Machine review of arXiv:2505.15501}
}

read the original abstract

We introduce the concept of protoknowledge to formalize and measure how sequences of tokens encoding Knowledge Graphs are internalized during pretraining and utilized at inference time by Large Language Models (LLMs). Indeed, LLMs have demonstrated the ability to memorize vast amounts of token sequences during pretraining, and a central open question is how they leverage this memorization as reusable knowledge through generalization. We then categorize protoknowledge into lexical, hierarchical, and topological forms, varying on the type of knowledge that needs to be activated. We measure protoknowledge through Knowledge Activation Tasks (KATs), analyzing its general properties such as semantic bias. We then investigate the impact of protoknowledge on Text-to-SPARQL performance by varying prompting strategies depending on input conditions. To this end, we adopt a novel analysis framework that assesses whether model predictions align with the successful activation of the relevant protoknowledge for each query. This methodology provides a practical tool to explore Semantic-Level Data Contamination and serves as an effective strategy for Closed-Pretraining models.

Figures

Figures reproduced from arXiv: 2505.15501 by the authors.

**Figure 1.** Protoknowledge Impact: LLMs acquire three protoknowledge forms from Knowledge Graphs, measured via Knowledge Activation Tasks (KATs). The degree of absorption of protoknowledge measured by KATs correlates with the Text-To-SPARQL performances of LLMs showing that, when protoknowledge is acquired, it is also positively used. However, the effectiveness of this emerging ability remains limited by the semantic bias in… view at source ↗

**Figure 2.** SPS scores for Wikidata and DBpedia. more effectively for highly popular items. This contributes to an observed overconfidence effect: performance is stronger when test data aligns with dominant semantic patterns present in the pretraining data. These findings emphasize the need to account for semantic distribution when designing KG-related evaluation benchmarks. Moreover, performance should be interpreted not only… view at source ↗

**Figure 3.** NO-URI vs Original Agreement and Disagreement. Above bar is reported Positive Agreement Ratio %. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: URI Recognition Task Task: Inverse Subsumption Using your knowledge of DBpedia ontology, provide the URI of the superclass for the resource [SUBCLASS]. Important: Your response must contain only the URI. Important: The URI must satisfy the triple: <[SUBCLASS], rdfs:sub…

**Figure 5.** Figure 5: Inverse Subsumption Task Task: Direct Subsumption Using your knowledge of DBpedia ontology, return only a list of URIs that are direct subclasses of CLASS. Important: URIs must be connected to CLASS by the property rdfs:subClassOf. Important: URIs must satisfy the trip…

**Figure 6.** Figure 6: Direct Subsumption Task Task: SV? Guess the OBJECT Considering your knowledge of DBpedia triples, can you fill the masked [MASKED_OBJECT] with an existing URI inside DBpedia? TRIPLE: {{’S’,’V’,[MASKED_OBJECT]}} Important: Answer only the URI! Do not invent URIs [PITH_…

**Figure 7.** Figure 7: SV? Task Task: ?VO Guess the SUBJECT Considering your knowledge of DBpedia triples, can you fill the masked [MASKED_SUBJECT] with an existing URI inside DBpedia? TRIPLE: {{[MASKED_SUBJECT],’V’,’O’}} Important: Answer only the URI! Do not invent URIs [PITH_FULL_IMAGE:f…

**Figure 8.** Figure 8: ?VO Task B Models and Hyperparameters To get a comprehensive evaluation, we use four different LLMs: GPT-4, GPT-3.5 (OpenAI, 2023), Llama-3.1-8B, Llama-3-8B, Llama3-70b-instruct (Grattafiori et al., 2024). We use greedy decoding in all experiments to ensure a more dete…

**Figure 9.** Figure 9: Original Approach Approach: No Label You are an expert in SPARQL and {KG_name}. Your task is to translate natural language questions into precise SPARQL queries that retrieve the desired information from {KG_name}. Guidelines: 1. Understand the input: Analyze the quest…

**Figure 11.** Figure 11: No URI approach 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

**Figure 13.** Figure 13: Wikidata vs DBpedia Positive Agreement in [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

**Figure 14.** Figure 14: Wikidata vs DBpedia Positive Agreement in [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 8 canonical work pages

[3]

Do not output anything else

Format the output: Enclose the SPARQL query within <SPARQL></SPARQL> tags. Do not output anything else. Question: {question} Entities: {URI} ({label}), ... Relations: {URI} ({label}), ... Query: Figure 9: Original Approach Approach: No Label You are an expert in SPARQL and{KG_name}. Your task is to translate natural language questions into precise SPARQL ...

work page
[8]

Do not output anything else

Format the output: Enclose the SPARQL query within <SPARQL></SPARQL> tags. Do not output anything else. Question: {question} Entities: {URI}, ... Relations: {URI}, ... Query: Figure 10: No Label Approach Approach: No URI You are an expert in SPARQL and{KG_name}. Your task is to translate natural language questions into precise SPARQL queries that retrieve...

work page
[9]

Understand the input: Analyze the question and use the provided Entities and Relations to construct the query

work page
[10]

Construct a valid SPARQL query: Use proper syntax and ensure the query retrieves accurate results from {KG_name}

work page
[11]

Do not output anything else

Format the output: Enclose the SPARQL query within <SPARQL></SPARQL> tags. Do not output anything else. Question: {question} Query: Figure 11: No URI approach 15 H Protoknowledge Analysis Impact H.1 Protoknowledge Analysis Framework in brief Given a form of KG Protoknowledge and a Text-to- SPARQL query instance, we perform the following steps:

work page
[12]

Extract a mini test set from the query, contain- ing relevant KG elements

work page
[13]

Evaluate protoknowledge on this mini set us- ing Knowledge Activation Tasks (KATs)

work page
[14]

H.2 Example of Framework Application on lexical form We report an example (Fig

If both the protoknowledge evaluation and the SPARQL generation are correct, the instance is marked as a Positive Agreement. H.2 Example of Framework Application on lexical form We report an example (Fig. 12) applying the frame- work for correlating lexical protoknowledge and Text-To-SPARQL inNo Label approach. Figure 12: Lexical protoknowledge Impact ana...

work page

Show all 10 references

[2023]

Preprint, arXiv:2202.07646

Quantifying memorization across neural lan- guage models. Preprint, arXiv:2202.07646. Jacopo D’Abramo, Andrea Zugarini, and Paolo Torroni

arXiv
[2025]

Perfect" satisfaction of the triple, while S represents

Investigating large language models for text- to-SPARQL generation. In Proceedings of the 4th International Workshop on Knowledge-Augmented Methods for Natural Language Processing, pages 66– 80, Albuquerque, New Mexico, USA. Association for Computational Linguistics. Chunyuan ...

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

Do not output anything else

Format the output: Enclose the SPARQL query within <SPARQL></SPARQL> tags. Do not output anything else. Question: {question} Entities: {URI} ({label}), ... Relations: {URI} ({label}), ... Query: Figure 9: Original Approach Approach: No Label You are an expert in SPARQL and{KG_name}. Your task is to translate natural language questions into precise SPARQL ...

work page

[2] [8]

Do not output anything else

Format the output: Enclose the SPARQL query within <SPARQL></SPARQL> tags. Do not output anything else. Question: {question} Entities: {URI}, ... Relations: {URI}, ... Query: Figure 10: No Label Approach Approach: No URI You are an expert in SPARQL and{KG_name}. Your task is to translate natural language questions into precise SPARQL queries that retrieve...

work page

[3] [9]

Understand the input: Analyze the question and use the provided Entities and Relations to construct the query

work page

[4] [10]

Construct a valid SPARQL query: Use proper syntax and ensure the query retrieves accurate results from {KG_name}

work page

[5] [11]

Do not output anything else

Format the output: Enclose the SPARQL query within <SPARQL></SPARQL> tags. Do not output anything else. Question: {question} Query: Figure 11: No URI approach 15 H Protoknowledge Analysis Impact H.1 Protoknowledge Analysis Framework in brief Given a form of KG Protoknowledge and a Text-to- SPARQL query instance, we perform the following steps:

work page

[6] [12]

Extract a mini test set from the query, contain- ing relevant KG elements

work page

[7] [13]

Evaluate protoknowledge on this mini set us- ing Knowledge Activation Tasks (KATs)

work page

[8] [14]

H.2 Example of Framework Application on lexical form We report an example (Fig

If both the protoknowledge evaluation and the SPARQL generation are correct, the instance is marked as a Positive Agreement. H.2 Example of Framework Application on lexical form We report an example (Fig. 12) applying the frame- work for correlating lexical protoknowledge and Text-To-SPARQL inNo Label approach. Figure 12: Lexical protoknowledge Impact ana...

work page