{"id":"0622af83-accf-4001-b87e-03a0364cda7e","arxiv_id":"2608.07700","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A researcher agent that edits its own text-to-SPARQL prompts and rules reaches 0.22 accuracy on DB25, with predicate grounding as the dominant error source.","lead":"The authors built a self-improving AI agent that rewrites its own prompts and rules between evaluation rounds to translate English questions into SPARQL queries over DBpedia. The best configuration reaches 0.22 accuracy on a validation set, and the paper argues that many remaining mistakes are benchmark ambiguities rather than agent failures.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run evaluation and the paper's own non-determinism admission undermine the 'converges quickly' and feature-impact claims; only the broad bottleneck pattern is robust.","rationale":"The reader's weakest assumption identifies precisely the load-bearing issue: all quantitative claims about self-improvement and feature effects rest on single-run measurements in a regime the paper itself describes as dominated by LLM non-determinism. My stress-test confirms this is the most vulnerable point of the central argument. The 'converges quickly' claim is not supported by the trajectory, which is non-monotonic and whose best values (0.22) are within plausible noise of the initial 0.20. The feature toggle analysis in Section 4.4 is further weakened by the explicit admission that metric gaps are due to non-determinism, making causal attribution impossible without repeated trials. The benchmark-ambiguity observation is supported by only one example, but that is a secondary claim; even if it were fully correct, the primary self-improvement result would still need repeated runs. I agree with the reader's conditional verdict: the paper presents a plausible architecture and a useful diagnostic pattern, but the headline claims require additional statistical support. The proposed concrete test—multiple seeds plus a control loop—would settle whether the observed trajectory reflects genuine self-improvement or merely selection noise. I do not see a reason to change the CONDITIONAL verdict; the paper should be accepted only if such evidence is added or the claims are softened accordingly.","tokens_in":8105,"tokens_out":5036,"duration_ms":48270,"concrete_test":"Run each of the nine agent versions for at least 10 independent trials on DB25 with varying sampling seeds, and report the mean and 95% confidence interval of overall accuracy for each version. Then perform a non-parametric pairwise test (e.g., Mann-Whitney U) between v1 and v6, and between v7 and v8. Also run a control condition where the researcher agent makes random or no edits for the same number of rounds. If the confidence intervals for v1 and v6 overlap and the control trajectory shows similar variance, then the convergence and feature-attribution claims should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that self-improvement converges quickly to 0.22 relies on a single run per agent version, and Section 4.4 explicitly states 'The gap in performance metrics is due to non-determinism in LLMs.' The accuracy series in Table 1 (0.20, 0.06, 0.18, 0.12, 0.18, 0.22, 0.17, 0.22, 0.21) is non-monotonic and oscillates within a range that is plausibly noise. In particular, v8 is described as a revert of v7 onto v6, yet it scores 0.22 while v7 scores 0.17; under the paper's own non-determinism caveat, this 0.05 gap cannot be attributed to the toggled 'numbered/structured rules' feature. Consequently, the feature-construction conclusions (e.g., the benefit of SELECT DISTINCT or the dbo: preference rule) are not identifiable from these data. Additionally, the researcher agent selects the best version using the same DB25 validation set on which the final 0.22 is reported, making that number a selected maximum rather than an unbiased estimate. The external DB26/CK26 results in Table 4 provide independent evidence that predicate grounding is difficult, but they use different backbones, datasets, and languages, so they do not validate the convergence trajectory or the specific rule attributions. The only robust finding is the consistent bottleneck in BGP-predicate accuracy (0.20-0.31 across versions).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a 'researcher agent' that iteratively edits the prompts, rules, and tool-orchestration code of a downstream text-to-SPARQL agent, using the 2025 DBpedia validation set (DB25) as its testbed. The agent evolves nine versions driven by DeepSeek v3.2; the best configuration reaches 0.22 exact-match accuracy on DB25, and the per-component breakdown indicates that basic-graph-pattern (BGP) predicate grounding is the weakest component. The paper also argues that several benchmark items penalize correct alternative queries because of property ambiguity in DBpedia, and it recommends a combined machine-translation/information-retrieval scoring scheme for future benchmarks.","tokens_in":8439,"tokens_out":5925,"duration_ms":52983,"significance":"If the empirical claims held, the paper would offer a useful demonstration of an LLM-driven outer loop for prompt and rule engineering in text-to-SPARQL, together with a robust diagnostic about predicate grounding being the bottleneck. Strengths include the open release of agent code and traces, the consistent bottleneck pattern across versions and backbones (BGP predicates 0.20-0.31 vs. inner operators 0.79-0.86 in Table 2), and the candid acknowledgment of LLM non-determinism. The contribution is, however, currently limited by single-run measurements on the same validation set used for model selection, and by a benchmark-ambiguity claim supported by only one worked example. The central convergence and feature-attribution claims therefore need additional empirical support before the paper's conclusions can be accepted.","major_comments":[{"comment":"The claim that self-improvement 'converges quickly' is not supported by the single-run trajectory. The reported accuracies are 0.20, 0.06, 0.18, 0.12, 0.18, 0.22, 0.17, 0.22, 0.21, which are non-monotonic and oscillate over a range that the paper itself attributes to LLM non-determinism ('The gap in performance metrics is due to non-determinism in LLMs', §4.4). With one run per version, the 0.05 difference between v7 (0.17) and v8 (0.22) cannot be attributed to the reverted feature, and the same applies to the other feature toggles. The paper should report multiple runs or confidence intervals and use a statistical test before drawing feature-construction or convergence conclusions.","section":"§4.4, Table 1"},{"comment":"The reported 0.22 is a selected maximum, not an unbiased performance estimate. The researcher agent selects the best configuration using DB25, and 0.22 is then reported on DB25; this makes the number a selection outcome rather than an estimate of expected performance. Furthermore, §4.6 states that 'positive trendlines suggest that the accuracy could have improved further beyond 0.22', which directly contradicts the abstract's 'converges quickly' assertion. The authors should report the selection process transparently and provide an unbiased evaluation on a held-out set or repeated runs to support the convergence claim.","section":"§4.4-§4.6"},{"comment":"The benchmark-ambiguity conclusion rests on a single hand-picked example (the science-fiction-novel question). The claim that 'several benchmark items' penalize correct queries needs a systematic analysis: how many missed items admit alternative SPARQL queries, how many of those alternatives return non-empty answers, and what fraction of the remaining errors they explain. Without such quantification, observation (iii) and the proposed change to a combined machine-translation/information-retrieval scoring scheme are not empirically grounded.","section":"§4.5"}],"minor_comments":[{"comment":"There is a typo: 'a singleinference loop' should read 'a single inference loop'.","section":"§3.1"},{"comment":"The statement 'The overall metric achieves 0.22 from v6 onwards' is contradicted by Table 1, where v7 scores 0.17 and v9 scores 0.21.","section":"§4.4"},{"comment":"The column names are inconsistent between Table 2 ('BGP Nodes', 'BGP Predicates', 'Inner ops', 'Outer ops') and Table 3 ('Nodes', 'Preds', 'Inner', 'Outer'); please harmonize the nomenclature.","section":"Tables 2 and 3"},{"comment":"Figure 1 is referenced in §4.6 but not included in the manuscript text; if it is intended as a figure, it should be provided, or the reference should be removed.","section":"Figure 1"},{"comment":"The relationship between 'DB25', 'DB26', and 'CK26' should be clarified earlier, since the paper moves from the First to the Second TEXT2SPARQL Challenge without stating the dataset correspondence explicitly.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as a challenge/workshop report, and the 'researcher agent' idea is a modest but reasonable engineering contribution. The main weakness is methodological: the validation-set selection, single-run evaluation, and anecdotal ambiguity analysis do not yet support the strong convergence and feature-attribution claims. I do not see a circularity problem, since the authors' self-citations [4,16] are not load-bearing for the empirical results. With repeated runs, honest reporting of the selection bias, and a systematic ambiguity analysis, the revised manuscript could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a useful empirical datapoint for the Text-to-SPARQL/challenge community, not a rigorous demonstration of self-improvement. The paper's real contribution is the nine-version feature-toggle study with per-component error breakdown, plus open code and traces. The consistent predicate-grounding bottleneck (0.20–0.31 across versions and backbones) is credible and is the result I'd trust most. The benchmark-ambiguity example (dbo:literaryGenre vs dbp:genre) is a real phenomenon and worth the field's attention.\n\nWhat the paper does well: the architecture is clearly described; Table 1 and Table 2 are transparent enough for a reader to see exactly what changed at each step; the authors admit non-determinism and that v8 reverted v7; the external CK26/DB26 results independently show predicate grounding remains hard. The citation pattern is fine—self-citations are to relevant prior work and not load-bearing for the main empirical claims.\n\nThe soft spots are the usual single-run problems, and they are load-bearing for the headline claims. The 'converges quickly' statement doesn't survive the trajectory (0.20, 0.06, 0.18, 0.12, 0.18, 0.22, 0.17, 0.22, 0.21). With one run per version and the paper's own admission that metric gaps are due to LLM non-determinism, the specific feature attributions (dbo: preference matters, SELECT DISTINCT matters, etc.) are not identifiable. And 0.22 is the best validation score selected by the researcher agent itself, so it is a selected maximum, not an unbiased accuracy estimate; the paper should say that plainly. The external results do not repair this because they involve different backbones, datasets, and languages. None of this destroys the empirical core: the bottleneck result and the critique of endpoint-only benchmark scoring remain standing. But the claims need rewording, and ideally repeated runs or confidence intervals.\n\nWho is this for: people building KGQA agents, and challenge organizers thinking about metrics and benchmark design. It deserves a serious referee, but my recommendation would be major revision: add multiple seeds or error bars, downgrade the convergence language, and present 0.22 as a selected max rather than a system accuracy.","headline":"Useful, transparent empirical datapoint for the Text-to-SPARQL community, but single-run evaluation undermines the convergence and feature-attribution claims.","tokens_in":8948,"tokens_out":3226,"would_cite":true,"duration_ms":31607,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A researcher agent that edits its own prompts, rules, and tool code converges to 0.22 overall accuracy on the 2025 DBpedia text-to-SPARQL validation set, with predicate grounding as the persistent bottleneck.","keywords":["Text-to-SPARQL","knowledge graph question answering","LLM agents","self-improving agents","DBpedia","predicate grounding","benchmark evaluation"],"falsifier":"Rerun each of the nine agent versions ten times on the same validation set with the same backbone and compare score distributions; if versions such as the v6 and v8 configurations are statistically indistinguishable, then the feature-attribution and convergence claims collapse. Alternatively, ask independent human annotators to judge the agent's alternative queries against DBpedia properties; if most \"missed\" queries are judged correct renderings, then exact-match benchmark scoring explains the 0.22 ceiling.","tokens_in":7910,"feed_emoji":"🤖","tokens_out":6303,"duration_ms":53669,"temperature":0.7,"pith_summary":"This paper tries to show that a \"researcher agent\"—an LLM that rewrites its own prompts, rules, and tool code after each evaluation round—can improve a text-to-SPARQL system on DBpedia without fine-tuning the model weights. On the 2025 DBpedia validation set, nine successive versions plateau at 0.22 overall accuracy, and the paper attributes the ceiling not to SPARQL syntax or modifiers but to basic-graph-pattern predicate selection. The study also claims that several remaining failures are benchmark artifacts: DBpedia has overlapping properties such as dbo:literaryGenre and dbp:genre, so correct queries can be marked wrong by exact match. If true, this would redirect effort from prompt engineering to entity and predicate grounding, and motivate structural scoring for text-to-SPARQL benchmarks.","feed_headline":"Self-improving agent plateaus at 0.22 on DBpedia text-to-SPARQL","feed_subtitle":"Residual errors sit in predicate selection, not SPARQL syntax; benchmark scoring may be the real ceiling.","key_machinery":"The central object is the researcher-agent loop. A downstream Text-to-SPARQL agent is given three tools—surface-form lookup, ontology lookup around a candidate entity, and a test-SPARQL executor—and iterates plan/ground/draft/verify steps. Wrapping it, the researcher agent edits the downstream agent's prompts, hard rules (e.g., \"prefer dbo: over dbp:\"), number of in-context examples, and small tool-orchestration pieces, producing a new version after each validation-set pass. The version trajectory and per-section scores carry the argument: toggling features such as SELECT DISTINCT, triple-count cues, rdf:type constraints, and the COUNT format changes overall accuracy, and the score pattern localizes the persistent failure to BGP predicates.","core_discovery":"The central claim is that an outer self-improvement loop, in which a researcher LLM proposes and tests edits to a downstream Text-to-SPARQL agent, converges quickly on the 2025 DBpedia validation set and then reaches 0.22 overall accuracy. In the feature-construction study, the best configuration ties for the highest overall accuracy while leading on both operator families; the per-section breakdown shows inner operators staying in the 0.79–0.86 band, outer operators rising to 0.86, BGP nodes settling near 0.59, and BGP predicates oscillating between 0.20 and 0.31. Because predicate scores remain the lowest throughout, the paper concludes the residual errors are dominated by relation grounding rather than query syntax or modifiers. A manual inspection supports a benchmark-side explanation: for a question such as \"How many unique authors have written science fiction novels?\", both the reference query and the agent's query are plausible renderings, because dbo:literaryGenre and dbp:genre coexist on overlapping but non-identical sets, and the expected query returns 2,892 results while the agent's alternative returns 1,116.","pith_inferences":["Because each agent version was evaluated only once and the paper itself notes that some performance gaps come from LLM non-determinism, the marginal gains credited to individual features should be treated as provisional until multi-seed evaluations are run.","The same outer-loop design could transfer to text-to-SQL and other structured-output tasks, where the hard part is also grounding surface terms to schema elements rather than producing valid syntax.","If benchmark scoring moved to structural metrics, the same agent could appear substantially more accurate than 0.22, since many rejected queries are plausible alternative renderings of the same question.","The positive trendlines in the version trajectory suggest that editing based on execution traces rather than hand-crafted meta-prompts might yield further gains, but the current single-run data cannot confirm that."],"forward_implications":["If the convergence claim holds, further prompt-level edits on the 2025 DBpedia validation set by this kind of researcher agent will not push accuracy much beyond 0.22; the remaining gains must come from better predicate and entity grounding.","Swapping the backbone LLM changes exact-match accuracy only modestly, with six models clustering between 0.15 and 0.23, so the reported ceiling is not primarily a matter of model capability.","The deployed configuration transfers to the newer DB26 benchmark with higher F1/NDCG values, which the paper attributes to improved annotation in the newer dataset, suggesting dataset quality is a major factor.","The proposed evaluation view—scoring text-to-SPARQL with a combination of machine translation and information retrieval metrics rather than exact endpoint match—would change how many \"misses\" are counted as errors."],"supporting_citations":[{"why":"Supplies DBpedia, the target knowledge graph whose ontology and endpoint are used throughout the experiments.","marker":"[1]"},{"why":"Defines the First International TEXT2SPARQL Challenge and the DB25 validation set with its per-component metrics.","marker":"[15]"},{"why":"Provides the ReAct reasoning-and-acting pattern that the downstream tool-using agent is built on.","marker":"[29]"},{"why":"The authors' prior fine-tuning approach, which the researcher-agent design replaces by editing source code instead of model weights.","marker":"[16]"},{"why":"Motivates treating SPARQL generation as a foreign-language translation task, the basis for the proposed MT/IR evaluation.","marker":"[4]"}],"fun_headline_variants":["Self-refining agent hits 0.22 on DBpedia, then stalls","Text-to-SPARQL bottleneck: predicate grounding, not syntax","Agent rewrites itself, but DBpedia predicates limit gains","DBpedia query agent plateaus at 0.22; relations are the wall","SPARQL self-improvement converges fast, then plateaus at 0.22"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that specific rule changes caused the accuracy differences rests on a single run per version, even though the paper itself says part of the gap is LLM non-determinism.","fun_headline_variants_meta":{"raw":{"variants":["Self-refining agent hits 0.22 on DBpedia, then stalls","Text-to-SPARQL bottleneck: predicate grounding, not syntax","Agent rewrites itself, but DBpedia predicates limit gains","DBpedia query agent plateaus at 0.22; relations are the wall","SPARQL self-improvement converges fast, then plateaus at 0.22"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000769,"raw_usage":{"total_tokens":3435,"prompt_tokens":1001,"completion_tokens":2434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":2334}},"tokens_in":617,"tokens_out":2434,"duration_ms":15606,"temperature":1.0,"reasoning_tokens":2334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:22:27.840832+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun each of the nine agent versions ten times on the same validation set with the same backbone and compare score distributions; if versions such as the v6 and v8 configurations are statistically indistinguishable, then the feature-attribution and convergence claims collapse. Alternatively, ask independent human annotators to judge the agent's alternative queries against DBpedia properties; if most \"missed\" queries are judged correct renderings, then exact-match benchmark scoring explains the 0.22 ceiling.","supporting_citations":[{"cited_title":"Lehmann, R","cited_arxiv_id":null,"evidence_quote":"Supplies DBpedia, the target knowledge graph whose ontology and endpoint are used throughout the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the First International TEXT2SPARQL Challenge and the DB25 validation set with its per-component metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ReAct reasoning-and-acting pattern that the downstream tool-using agent is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The authors' prior fine-tuning approach, which the researcher-agent design replaces by editing source code instead of model weights."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates treating SPARQL generation as a foreign-language translation task, the basis for the proposed MT/IR evaluation."}],"review_version":1}