{"id":"8395bef7-9669-40e3-bae9-dd11d955615a","arxiv_id":"2505.15501","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new 'protoknowledge' analysis links lexical, hierarchical, and topological knowledge graph recall in LLMs to Text-to-SPARQL performance, offering a black-box route to detect semantic contamination.","lead":"This paper defines protoknowledge, a way to measure how much language models internalize knowledge-graph facts during pretraining, using three probe task families. It then correlates these measurements with Text-to-SPARQL accuracy under different prompt conditions and proposes the correlation as a tool for detecting semantic data contamination in closed models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Agreement-vs-Disagreement in Figure 3 is not evidence of protoknowledge dependence: under No URI's low success rates, independent success/failure already predicts Agreement > Disagreement, so the reported contrast is a base-rate artifact.","rationale":"The reader correctly identified that the Positive Agreement framework assumes query success is mediated by protoknowledge activation, and that the Original-prompt data showing Disagreement dominate violate that assumption. My concern is adjacent but more fundamental: even if the mediation assumption were true, the specific statistic used to support the claim (Agreement vs Disagreement in No URI) is uninformative because it does not control for the low base rates of both events. When both query generation and triple completion are rare, Agreement is expected to exceed Disagreement under complete independence, so the observed pattern in Figure 3 cannot distinguish dependence from a base-rate effect. This directly undermines the paper's headline inference that protoknowledge 'shapes' Text-to-SPARQL behaviour and the derived claim of a practical semantic-contamination detection tool. I do not object to the descriptive KAT measurements or the semantic-bias observations, which are reasonable and could stand on their own. The issue is specifically the inferential step in Section 5.1. A straightforward reanalysis with contingency tables and an independence baseline would settle it. I therefore keep the reader's CONDITIONAL verdict unchanged; if the reanalysis fails to show significant positive dependence, the central claim should be revised or weakened.","tokens_in":14551,"tokens_out":8347,"duration_ms":78699,"concrete_test":"Recompute Figure 3 as a 2x2 contingency table per model, per setting (Original/No Label/No URI, Wikidata/DBpedia), with rows = SPS correct/incorrect and columns = query correct/incorrect. From the observed marginals, compute the agreement expected under independence, E = p_correct_query*p_correct_SPS + (1-p_correct_query)*(1-p_correct_SPS), and compare it with observed agreement using a Fisher exact test or Cohen's kappa. If observed agreement does not significantly exceed the independence baseline, or if kappa is near zero, the central claim in Section 5.1 is not supported by the data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing evidence for the paper's central claim is the Section 5.1 / Figure 3 observation that, in the No URI setting, Agreement (query correct iff SPS correct) consistently exceeds Disagreement. This statistic conflates true dependence with marginal success rates. If query success has probability p and SPS success has probability q, then under independence P(Agreement) = pq + (1-p)(1-q) and P(Disagreement) = p(1-q) + (1-p)q. For the small success rates observed in No URI (Table 4 F1 scores of 3.7-29.1%; Table 8 SPS perfect scores mostly below 30%), the independence model already predicts Agreement to dominate; for example, p=0.1, q=0.2 gives 0.74 Agreement versus 0.26 Disagreement. The reverse pattern in the Original setting, where query success is much higher, is likewise compatible with independence. Consequently, the plotted Agreement/Disagreement comparison carries no information about whether triple completion is statistically or causally related to query generation. Without a dependence measure that controls for marginals, the claim that topological protoknowledge 'shapes' Text-to-SPARQL, and the proposed semantic-contamination tool built on it, are unsupported by the reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the notion of 'protoknowledge' to describe how LLMs internalize Knowledge Graph content during pretraining and reuse it in downstream tasks. It distinguishes lexical, hierarchical, and topological forms, measures them through Knowledge Activation Tasks (URI recognition, subsumption, and triple completion), and analyzes their dependence on item popularity as evidence of semantic bias. It then examines Text-to-SPARQL performance under Original, No Label, and No URI prompting conditions and uses a Positive Agreement framework to claim that success in triple completion predicts success in query generation, particularly when URI hints are absent. The paper concludes that this framework offers a practical tool for detecting semantic-level data contamination in closed-pretraining models.","tokens_in":14876,"tokens_out":7837,"duration_ms":69485,"significance":"If the central claim were supported, the paper would provide a useful black-box method for linking internal KG recall to structured query generation and for probing contamination in models whose pretraining data is unavailable. The taxonomy of lexical, hierarchical, and topological protoknowledge and the KAT design are a reasonable starting point for such an analysis. However, the evidence as presented does not establish the claimed dependence: the main Agreement/Disagreement statistic is confounded by base rates, and the paper's own assumption about query success implying protoknowledge activation is contradicted by its Original-mode results. A reanalysis with margin-controlled dependence measures could make the contribution meaningful, but the current support is inadequate.","major_comments":[{"comment":"The central evidence that 'Agreement consistently exceeds Disagreement' in the No URI setting is confounded by the marginal success rates. Under independence of query success (probability p) and SPS success (probability q), P(Agreement) = pq + (1-p)(1-q) and P(Disagreement) = p(1-q) + (1-p)q. In the No URI setting, Table 4 reports F1 values of 3.67–29.09 and Table 8 reports perfect SPS values mostly below 30%; for example, p=0.1 and q=0.2 already predict 74% Agreement against 26% Disagreement. The plotted contrast therefore carries no information about whether triple completion and query generation are statistically or causally related. The paper should report a margin-controlled statistic (e.g., Cohen's kappa, a chi-square test on the 2×2 table, or a permutation baseline) and should compare the observed Agreement against the independence prediction rather than against Disagreement alone.","section":"§5.1 and Figure 3"},{"comment":"The framework assumes that 'a correct SPARQL query implies successful activation of protoknowledge', and for topological protoknowledge it assumes the reverse as well. This is directly contradicted by the same figure: in the Original setting Disagreement dominates, which must include cases where the SPARQL query is correct while the corresponding triple completion is incorrect. If correct queries can be produced without successful triple recall, then query performance does not require the measured protoknowledge, and the Positive Agreement ratio is merely a conditional frequency rather than evidence of mediation. The paper must report the full 2×2 contingency table (query correct/incorrect by SPS correct/incorrect) and separate the two directional cells of Disagreement.","section":"§5.1, Figure 3, Original setting"},{"comment":"Equation (1) defines SPS as |Tpredicted ∩ TQ| / |TQ|, where TQ is 'all entity-property pairs relevant to Qgold' and Tpredicted comes from the SV? and ?VO tasks. The Limitations paragraph states that the S?O task is deliberately omitted. This makes the metric's coverage unclear: if TQ contains triples whose property must be inferred from a subject-object pair, those triples can never appear in Tpredicted, so SPS will be systematically depressed and the Agreement analysis will be distorted. The paper should either restrict TQ to entity-property pairs testable by SV? and ?VO or report the proportion of Qgold triples covered by these two tasks.","section":"§4.4 and Limitations"},{"comment":"The quantitative evidence for the popularity trend is not auditable in its current form. In the LF row for entities, the cell '3.11% (5/9)' is internally inconsistent: 5/9 is 55.6%, while 5/161 is 3.11%. Several other cells show the same mismatch between percentage and parenthetical ratio. Additionally, the hierarchical results in Tables 2 and 3 rest on supports as small as 2–5 items and are reported without any significance or confidence intervals. The semantic-bias claims need exact counts, internally consistent percentages, and at least exact binomial confidence intervals or an explicitly stated exploratory status.","section":"§4.2, Table 1"}],"minor_comments":[{"comment":"The lexical Positive Agreement analysis excludes items below the 50th percentile of popularity without justification, and Table 12's explanation of the threshold is unclear; the paper should state the selection rule explicitly and, if possible, also report results on the full set.","section":"§5.1, lexical PA analysis"},{"comment":"TQ and Tpredicted are not formally defined as sets, and the role of Perfect versus Soft matches in the SPS score is described only in prose; a precise set-theoretic definition would improve reproducibility.","section":"Equation (1)"},{"comment":"The bars do not show the number of examples behind each percentage, even though the text notes that some ratios rest on very small counts (e.g., three examples for Llama-3_70B); adding per-bar n values is essential for interpreting the figure.","section":"Figure 3"},{"comment":"The text states Llama-3_70B has hierarchical PA of 32/39, while Table 6 lists 32/40; these should be harmonized.","section":"Section 5.1 and Table 6"},{"comment":"There are several typos and minor wording issues, including 'categorizeprotoknowledge' in the abstract, 'hypotesize' in Section 5.1, 'benchmkarks' and 'Hierachical' in the Limitations paragraph, and the nonstandard use of 'overconfidence' in Section 4.5 to mean accuracy bias rather than confidence calibration.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main reason for choosing major_revision rather than reject is that the central confound is, in principle, fixable: the authors already possess the per-query SPS and Text-to-SPARQL outcomes, so they can recompute the analysis with margin-controlled dependence measures and report the directional contingency cells. If that reanalysis supports the claim, the framework could be a useful contribution; if it does not, the contamination-tool framing should be removed. The current manuscript, however, does not support the central claim as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper gives a clean way to think about three levels of KG knowledge (lexical, hierarchical, topological) and a coherent set of probes. The KATs are sensible, the popularity effect is consistent across tasks, and the limitations section is honest. But the load-bearing evidence for the headline claim — that protoknowledge 'shapes' Text-to-SPARQL and can expose semantic contamination — does not hold up. The Section 5.1 / Figure 3 comparison of Agreement vs Disagreement conflates true dependence with marginal success rates. Under independence, when both query success and SPS success are low (as in No URI), Agreement dominates; the paper's own numbers (e.g., query F1 of 3.7–29.1%, SPS perfect scores mostly below 30%) already predict that. No dependence measure controlling for marginals is reported, so the plotted contrast carries no information about whether triple completion and query generation are related. The paper's Original condition, where Disagreement dominates, also contradicts the stated assumption that a correct SPARQL query implies correct topological protoknowledge. That assumption is doing real work, and the data refute it.\n\nWhat's new and worth keeping: the protoknowledge taxonomy is a reasonable organizing concept, and applying it to Text-to-SPARQL contamination is new. The URI recognition, subsumption, and triple-completion tasks are a coherent probe suite. The popularity/semantic-bias result is plausible and appears across several tasks, though it is mostly descriptive.\n\nSoft spots in proportion: small samples (2–13 examples per cell in places), no significance tests, post hoc popularity splits, and the lexical PA analysis discards below-median items without justification. Table 1 has inconsistent percentages and counts. None of these kill the framework; they just mean the quantitative claims are fragile.\n\nBottom line: as a methodology proposal, the paper deserves referee time. As a demonstration that protoknowledge shapes downstream behavior, it needs a proper dependence measure (e.g., compare Agreement to the independence baseline) and ideally a validation against known contamination. If the authors redo the analysis that way, the framework could become solid. Send it to review, but with a clear request to fix the base-rate problem.","headline":"Useful probe framework, but the Section 5.1 Agreement/Disagreement contrast is a base-rate artifact — the central 'shapes behaviour' claim needs a marginals-controlled dependence measure.","tokens_in":15360,"tokens_out":2045,"would_cite":false,"duration_ms":17968,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a model's success at recalling knowledge-graph triples—topological protoknowledge—determines whether it can produce correct Text-to-SPARQL queries when no URI hints are given, and that this agreement exposes…","keywords":["protoknowledge","knowledge graphs","LLM memorization","generalization","Text-to-SPARQL","Knowledge Activation Tasks","semantic bias","data contamination"],"falsifier":"Take a structurally identical set of questions to QALD-9 but replace all entity and relation URIs with random, semantically uninformative identifiers, and run the No-URI condition. If a model with near-zero SPS still produces correct queries at non-trivial rates, or if Agreement no longer exceeds Disagreement, the claimed dependence on memorized protoknowledge would be refuted.","tokens_in":14369,"feed_emoji":"🧠","tokens_out":8694,"duration_ms":69075,"temperature":0.7,"pith_summary":"The paper introduces a concept it calls protoknowledge: the structured knowledge a language model stores about a knowledge graph during pretraining, at three levels—lexical (label-to-URI mappings), hierarchical (class-subclass relations), and topological (which triples actually hold). It tries to establish that these levels can be measured separately with simple recall tasks and that the measured level predicts downstream behavior: in Text-to-SPARQL, models that correctly complete the relevant triples are the ones that generate correct queries when no URI hints are supplied. The paper's headline evidence is that, in the No-URI setting, agreement between correct triple completion and correct query generation consistently outweighs disagreement across models and graphs. If the claim holds, it gives researchers a black-box way to infer whether a closed model is recalling memorized semantic content—semantic-level data contamination—rather than reasoning from context. A careful reader should care because contamination audits for models with undisclosed training data are otherwise hard to perform.","feed_headline":"Triple recall predicts SPARQL writing when URI hints vanish","feed_subtitle":"Protoknowledge tests show query success tracks triple recall, giving a black-box contamination probe.","key_machinery":"The machinery is a per-example correlation between two measurements. The first is a set of Knowledge Activation Tasks (KATs), each designed to isolate one protoknowledge form: URI recognition (lexical), direct and inverse subsumption (hierarchical), and SV?/?VO triple completion (topological). The second is the Speculative Protoknowledge for SPARQL (SPS) score, defined as $SPS = |T_{\\text{predicted}} \\cap T_{Q}| / |T_{Q}|$, which measures how many entity-property pairs from a gold query the model can reconstruct as valid triples, verified by SPARQL ASK queries. These two measurements are then combined in a Positive Agreement/Disagreement framework that classifies each query by whether correct query generation and correct protoknowledge activation coincide, which is what lets the paper attribute query success to memorized structure.","core_discovery":"On the paper's own terms, the central discovery is that LLMs absorb knowledge-graph content as reusable protoknowledge in three distinguishable forms, and that the activation of each form can be measured by Knowledge Activation Tasks (KATs): label-to-URI recognition for lexical, direct and inverse subsumption for hierarchical, and subject-verb-object triple completion for topological protoknowledge. The paper's central result is that these measurements align with Text-to-SPARQL behavior: in the No-URI condition, successful query generation and successful triple completion agree more often than they disagree, and Positive Agreement—correct queries accompanied by correct protoknowledge activation—often reaches high ratios, including 100% for GPT-3.5 Turbo. The same alignment is present, though weaker, for lexical protoknowledge in the No-Label condition and for hierarchical protoknowledge on DBpedia. The paper presents this query-by-query agreement as a practical tool for exposing semantic-level data contamination in closed-pretraining models.","pith_inferences":["The same query-by-query agreement framework could be adapted to other structured tasks, such as Text-to-SQL, by defining schema protoknowledge through recall tasks over tables, columns, and foreign-key triples.","If protoknowledge is frequency-driven, contamination audits should stratify Positive Agreement by entity popularity; a sharp drop on rare entities would indicate memorization of common web content rather than robust generalization.","A testable extension the paper does not run: rerun the No-URI condition with entity and relation URIs replaced by random strings; a collapse in Agreement would confirm that success depends on memorized token-level structure rather than abstract graph reasoning.","The paper's deliberate omission of the S?O triple-completion task suggests a full three-slot SPS would be an even stronger predictor, provided overly frequent properties are controlled for."],"forward_implications":["In the No-URI condition, a model's Text-to-SPARQL performance on QALD-9 can be anticipated from its SPS triple-completion scores: models that fail to recall a query's triples will generally fail to produce the query.","In the Original condition, where URIs and labels are supplied, correct query generation no longer requires protoknowledge, so benchmarks that provide full context underestimate the role of memorized structure.","Lexical protoknowledge (label-to-URI mapping) is what lets models keep working in the No-Label condition, especially on Wikidata's non-human-readable identifiers.","The per-example Positive Agreement ratio offers a black-box profile of semantic-level data contamination for closed-pretraining models, complementing verbatim-memorization checks.","Because protoknowledge is stronger for popular items, performance on frequent entities and properties should not be extrapolated to rare or out-of-distribution content."],"supporting_citations":[{"why":"Supplies the QALD-9 Plus Text-to-SPARQL benchmark in Wikidata and DBpedia versions that the paper's own experiments and SPS test sets are built on.","marker":"(Perevalov et al., 2022)"},{"why":"Provides the zero-shot baseline prompt that defines the Original, No-Label, and No-URI conditions.","marker":"(D'Abramo et al., 2025)"},{"why":"Underpins the hypothesis that triple-level knowledge acquired in pretraining supports downstream structured-query tasks.","marker":"(Moiseev et al., 2022)"},{"why":"Defines the Llama-3 family of models whose protoknowledge and query performance are measured.","marker":"(Grattafiori et al., 2024)"},{"why":"Defines the GPT-4 and GPT-3.5-Turbo closed-pretraining models used in the evaluations.","marker":"(OpenAI, 2023)"},{"why":"Establishes the memorization capacity of LLMs that protoknowledge builds on.","marker":"(Carlini et al., 2023)"},{"why":"Motivates the black-box contamination-analysis strategy that the Positive Agreement framework extends to knowledge graphs.","marker":"(Ranaldi et al., 2024)"},{"why":"Provides prior label-prediction tasks over DBpedia triples that the triple-completion KATs are modeled on.","marker":"(Lo et al., 2023)"}],"fun_headline_variants":["Protoknowledge: how LLMs tap memorized graph data for SPARQL","Triple recall matches SPARQL success, exposing contamination","KG protoknowledge drives LLM query writing","Memorized graph knowledge predicts SPARQL generation","LLM SPARQL skill ties to memorized KG triple recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a correct SPARQL query implies the model actually recalled the relevant knowledge-graph triples (and, for topological protoknowledge, that correct recall is required for a correct query); if query success can come from surface pattern-matching or from copying URIs in the prompt, the agreement statistics do not measure protoknowledge.","fun_headline_variants_meta":{"raw":{"variants":["Protoknowledge: how LLMs tap memorized graph data for SPARQL","Triple recall matches SPARQL success, exposing contamination","KG protoknowledge drives LLM query writing","Memorized graph knowledge predicts SPARQL generation","LLM SPARQL skill ties to memorized KG triple recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00082,"raw_usage":{"total_tokens":3586,"prompt_tokens":939,"completion_tokens":2647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2560}},"tokens_in":555,"tokens_out":2647,"duration_ms":17483,"temperature":1.0,"reasoning_tokens":2560,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:16:05.080108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a structurally identical set of questions to QALD-9 but replace all entity and relation URIs with random, semantically uninformative identifiers, and run the No-URI condition. If a model with near-zero SPS still produces correct queries at non-trivial rates, or if Agreement no longer exceeds Disagreement, the claimed dependence on memorized protoknowledge would be refuted.","supporting_citations":[],"review_version":1}