{"id":"07d399ce-5941-46d9-9b72-964b89d50a70","arxiv_id":"2505.05118","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Schema filtering, especially exact-match pruning, reduces prompt length and cost for Text2Cypher and improves accuracy for smaller models, though larger models gain less.","lead":"This paper tests five ways of trimming the database schema shown to an AI when it writes Cypher graph queries. The best trimming shortened prompts, cut costs, and helped smaller models, but larger models lost little and sometimes preferred the longer prompts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cost-reduction claim omits the per-query overhead of dynamic schema pruning; Table 3 counts only input tokens, so \"reduced cost for all\" is not yet supported end-to-end.","rationale":"The reader's weakest assumption concerned generalization from demo-oriented simple schemas to complex real-world schemas. That is a valid external-validity concern and is acknowledged in the paper's Section 5. However, I consider the more load-bearing weakness to be internal to the cost analysis: the quantitative cost claim is built on a table that deliberately ignores the computational cost of the very filtering operations being proposed. The paper says the overhead is minimal but supplies no measurement, so the central claim that filtering \"reduced cost for all of those we tested\" is not fully established even for the tested subset. This does not require rejecting the paper; the claim is plausible and prior work supports token savings. It does require an additional condition: the authors should report end-to-end cost or at least measure the pruning overhead. Since the reader's verdict is already CONDITIONAL, my recommendation is UNCHANGED rather than a move to a different verdict. The main adjustment would be to make the missing overhead measurement an explicit condition of acceptance and to phrase the cost conclusion more narrowly as \"input-token cost\" unless end-to-end evidence is provided.","tokens_in":7577,"tokens_out":3785,"duration_ms":42133,"concrete_test":"Benchmark end-to-end latency and total cost for all 2,471 test queries on the same hardware/API across the five schema formats, measuring schema extraction and pruning time (regex, NER, embedding similarity), generation time, and allowing output token counts to vary naturally. Recompute the cost comparison including this overhead. If Pruned by Exact-Match total cost per query is not below Base Schema, the cost-reduction conclusion should be weakened or restricted to input-token savings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central cost claim in Section 4.2.1 and the conclusion is that schema filtering \"reduced cost for all of those we tested.\" Table 3 computes costs from median token counts under the explicit assumption that \"output lengths remain constant and only input tokens contribute to the cost.\" This ignores the compute cost of the dynamic pruning itself. Unlike the cached static schemas, the three dynamic methods run per query: regex matching for exact-match, spaCy NER for NER-masked pruning, and embedding-based similarity scoring for similarity pruning. The paper asserts that this overhead is \"minimal,\" especially for exact-match, but provides no measurement of latency, CPU/GPU time, or additional API calls. For a self-hosted model billed by wall-clock GPU time (as in the LLaMA-3.1-8B row), even a few seconds of extra preprocessing per query can offset the token savings. For API-based deployment, the same issue applies if preprocessing runs on paid infrastructure. Because cost reduction is one of the two headline contributions, this unmeasured overhead is load-bearing: if end-to-end cost including filtering does not remain below the static-schema baseline, the main practical recommendation is unsupported. This is an internal incompleteness rather than a disagreement with prior work. The performance generalization concern raised by the reader is real, but the cost claim can fail even on the tested dataset if overhead is included.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies schema filtering for Text2Cypher, the task of translating natural language questions into Cypher graph queries. The authors compare two static schema formats (Enhanced Schema and Base Schema from Neo4j) with three dynamic pruning methods (exact-match pruning, NER-masked exact-match pruning, and embedding-similarity pruning) using a public Text2Cypher dataset. They report token-length statistics, translation-based (Google-BLEU) and execution-based (exact-match) accuracy for Llama-3.1-8B, Qwen2.5-7B, and Gemini-1.5-Flash, and a cost model based on input token counts. The main claims are that exact-match pruning yields the highest accuracy on their tested setup, that reducing schema size improves performance for most models, and that schema filtering reduces cost for all tested models. The paper also acknowledges limitations: the dataset consists of demo-oriented databases with relatively simple schemas, and the methods are heuristic.","tokens_in":7840,"tokens_out":3025,"duration_ms":30341,"significance":"If the results hold, the paper provides a practical and low-cost recipe for improving Text2Cypher performance with smaller models and for reducing token expenditures across models. The exact-match pruning method is simple, easy to implement, and the empirical comparison across static and dynamic schemas on a large public dataset is a useful contribution for practitioners. The use of both lexical and execution-based metrics is a strength, and the paper is transparent about the dataset subset and the heuristic nature of the filtering. However, the central cost claim is not fully supported because the overhead of dynamic pruning is not measured, and the performance claims lack error bars or significance testing. The contribution is therefore incremental but potentially useful if the open issues are addressed.","major_comments":[{"comment":"The cost-reduction claim in Section 4.2.1 and the conclusion (\"reduced cost for all of those we tested\") is based on Table 3, which counts only input tokens under the assumption that \"output lengths remain constant and only input tokens contribute to the cost.\" This ignores the per-query compute overhead of the dynamic pruning methods: exact-match requires regex processing, NER masking requires spaCy inference, and similarity pruning requires embedding computation. The paper asserts in Section 4.2.1 that this overhead is \"minimal,\" but no latency, CPU/GPU time, or additional API cost is measured. For self-hosted models billed by wall-clock GPU time, or for API-based deployments where preprocessing runs on paid infrastructure, the token savings may not translate into end-to-end cost savings. The cost claim is load-bearing for the paper's main practical recommendation, so it should be either measured end-to-end or explicitly qualified as token-cost-only.","section":"Section 4.2.1, Table 3"},{"comment":"The performance claims, including \"The highest accuracy is achieved with the 'Pruned by Exact-Match Schema'\" and \"reducing schema size improved performance for most models,\" are based on single evaluation runs without error bars, confidence intervals, or significance tests. Given that models can be sensitive to sampling temperature and other sources of randomness, the observed differences (some around 2-5%) may not be robust. The paper should either run multiple independent trials and report variance, or present the raw scores with a clear statement that the comparison is anecdotal. This is load-bearing because the main empirical contribution is that pruning improves or preserves accuracy for most models.","section":"Section 4.2.2, Figures 5 and 6"},{"comment":"The similarity-based pruning method is not reproducible as described: the paper does not specify the similarity threshold, the embedding model used (beyond \"spaCy framework\" in Section 4.1), or the exact similarity measure. These are free parameters that directly control how many schema elements are retained and thus affect both token length and performance. The paper should report these details, and ideally include a sensitivity analysis over the threshold. This is load-bearing for the method's reproducibility and for interpreting the token-distribution results in Table 2.","section":"Section 3.2, Pruned by Similarity"},{"comment":"The abstract and conclusion make general claims such as \"schema filtering effectively optimizes Text2Cypher\" and \"reducing schema size improved performance for most models,\" while the paper's own limitation statement in Section 5 notes that the evaluation used \"demo-oriented databases\" with the longest schema around 2700 tokens, which are simpler than real-world schemas. This is a scope mismatch: the central claim is broader than the evidence provided. The authors should either qualify the claims to explicitly reference the tested schema-complexity range, or add a discussion of why the findings are expected to transfer to more complex schemas. This is load-bearing because the practical value of the paper depends on generalization beyond the demo databases.","section":"Section 5, Conclusion and Abstract"}],"minor_comments":[{"comment":"The user instruction prompt in Table 1 contains a placeholder {schema} but the system instruction mentions \"Use only the provided relationship types and properties in the schema.\" It would be clearer to also include the model-specific output format constraints (e.g., no leading whitespace or markdown) that the post-processing step in Section 4.1 addresses.","section":"Section 1, Table 1"},{"comment":"The text says \"This approach compares node labels, relationship types, and properties to words in the input question\" and then mentions \"exact case-insensitive match.\" It would be helpful to specify whether stemming or lemmatization is applied, and whether matching is done on whole tokens or substrings.","section":"Section 3.2, Pruned By Exact-Match"},{"comment":"The paper refers to \"GoogleAIStudio/Gemini-1.5-Flash\" but the Gemini model is accessed via an API; the paper should specify the API configuration (e.g., temperature, top-p, max tokens) to ensure reproducibility.","section":"Section 4.1, Models"},{"comment":"Table 3 uses \"20K instances\" in the cost scenario, while the test set is 2,471 samples. The scenario is clearly labelled, but it is worth noting explicit that the cost table is a hypothetical scale-up and not the actual experimental cost.","section":"Section 4.2.1, Table 3"},{"comment":"There is a typographical artifact in the author affiliation: \"/envel⌢pe-⌢penmakbule.ozsoy@neo4j.com\" should be cleaned up, and the spaCy library should be cited properly.","section":"References and author affiliation"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style empirical study with a modest but potentially useful contribution. The main issue is that the headline cost-reduction claim is not supported because the dynamic-pruning overhead is unmeasured; this is fixable with additional experiments or a narrowed claim. The performance claims also need robustness checks. The scope limitation in Section 5 should be reflected in the abstract and conclusion. I would be comfortable with acceptance after these revisions, provided the cost claim is either measured or explicitly limited to token costs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a modest, honest empirical study of schema filtering for Text2Cypher. What is actually new is a systematic comparison of two static and three dynamic schema-pruning methods on one public dataset, with an analysis of token distributions and a rough cost model. The finding that exact-match pruning helps smaller models and that larger models benefit less is consistent with prior Text2SQL and Text2GQL work, and the author says so. The paper does not oversell novelty: it cites Liang et al. and NAT-NL2GQL, positions the contribution as an extension, and openly lists limitations.\n\nWhat it does well: the experimental design is clear, the two static schemas are well described, and the token-distribution analysis gives a concrete sense of how much pruning can shrink prompts. The cost table, while simplified, makes the token savings tangible. The performance evaluation uses both lexical and execution-based metrics, which is the right thing to do for query generation.\n\nThe soft spots are real but not fatal. The biggest one, which the stress-test note correctly identifies, is the cost claim. The paper says dynamic pruning reduces cost for all methods tested, but Table 3 counts only input tokens under the assumption that output lengths remain constant and ignores the compute cost of running regex, spaCy NER, and similarity scoring per query. For the self-hosted LLaMA row, that overhead could be substantial relative to the token savings. The paper acknowledges the overhead exists but dismisses it as minimal without measuring it. Since cost reduction is a headline contribution, this is a load-bearing gap. A few latency numbers or a breakdown of preprocessing time would fix it.\n\nSecond, all results are single runs with no error bars, significance tests, or standard deviations. For LLM generation, this matters. The performance differences between schema formats are small in some cases (2–5% for Qwen and Gemini), and without variance we cannot tell if they are noise. Third, the similarity threshold and the spaCy embedding model are not specified, which hurts reproducibility. Fourth, the datasets are demo-oriented with simple schemas; the author acknowledges this, and the largest schema is around 2,700 tokens. That does not invalidate the conclusions, but it limits generalization.\n\nThe citation pattern is fine. The author cites the relevant Text2SQL and Text2GQL schema-linking literature and her own prior dataset paper, which is appropriate. I do not see circular reasoning or invented entities.\n\nWho is this for? Practitioners building Text2Cypher systems on moderately sized schemas, and researchers looking for a benchmark comparison of cheap filtering heuristics. It is not a research-landscape-changer, but it is a legitimate incremental contribution with transparent limitations.\n\nRecommendation: I would send it to peer review for the workshop it is targeting. It deserves a serious referee, but the referee should ask for overhead measurements, error bars, and the missing parameters. As it stands I would not fully endorse the cost-reduction claim as an end-to-end result until the filtering overhead is accounted for.","headline":"Useful practical study on schema filtering for Text2Cypher, but the headline cost claim omits the overhead of the dynamic filtering itself, and there are no error bars.","tokens_in":8351,"tokens_out":1251,"would_cite":true,"duration_ms":14076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Schema pruning makes Text2Cypher cheaper and at least as accurate for most models tested.","keywords":["Text2Cypher","schema filtering","schema linking","large language models","cypher query generation","prompt token length","cost reduction"],"falsifier":"Take a real-world knowledge graph with thousands of node labels and relationship types, build a test set with naturally phrased questions that do not literally repeat schema element names, and measure execution-based exact-match accuracy for Llama-3.1-8B under the five schema formats. If exact-match pruning performs worse than the full Base Schema on that test set, the paper's central claim would fail for the regime it is meant to generalize to.","tokens_in":7383,"feed_emoji":"🕸️","tokens_out":1672,"duration_ms":18672,"temperature":0.7,"pith_summary":"This paper asks whether, when an LLM translates a natural-language question into a Cypher query, it is better to hand the model the full graph-database schema or only the schema elements that are relevant to that question. The author proposes three dynamic pruning methods, alongside two static full-schema formats, and tests them on a public Text2Cypher benchmark. The central finding is that reducing schema size improves or maintains query-generation accuracy for most tested models while consistently lowering token cost. If the finding holds beyond the benchmark, it means that prompt engineering for graph queries should treat the schema not as a fixed appendix but as a per-question resource to be filtered.","feed_headline":"Schema pruning makes Text2Cypher cheaper and more accurate","feed_subtitle":"Exact-match schema filtering cuts prompt tokens dramatically while improving accuracy on smaller LLMs.","key_machinery":"The central mechanism is the schema-pruning step inserted between the natural-language question and the Cypher prompt. The paper compares two static schemas (Enhanced Schema, which includes example property values, and Base Schema, which does not) with three dynamic schemas: 'Pruned by Exact-Match,' which retains schema elements whose node labels, relationship types, or properties appear as case-insensitive exact matches in the question; 'NER Masked & Pruned by Exact-Match,' which first replaces named entities with their entity types to avoid semantically irrelevant string matches; and 'Pruned by Similarity,' which keeps elements whose embedding-based similarity to question terms exceeds a threshold. The pruning step is what determines prompt length, and prompt length in turn drives both model accuracy and token cost.","core_discovery":"The paper's central claim is that dynamically pruning the database schema before feeding it to an LLM is both a performance-improving and cost-reducing strategy for Text2Cypher. Among the five schema formats evaluated, the 'Pruned by Exact-Match Schema' produced the highest lexical accuracy (Google-BLEU) and the highest execution-based exact-match accuracy for the Llama-3.1-8B model, while cutting the 95th-percentile prompt token length from roughly 2,700 tokens to roughly 529 tokens. For smaller models, shorter prompts clearly improved both lexical and execution-based translation quality. For Gemini-1.5-Flash, the larger model with a long context window, longer prompts delivered slightly better results, but the drop from pruning remained below five percent, and cost savings applied to every model tested.","pith_inferences":["The paper's exact-match pruning result hints at a broader principle for all schema-linked text-to-query tasks: the marginal value of a schema token to an LLM may be negative when it competes with the question itself for attention, and positive only when it carries information the model cannot infer from context. That principle, if true, would also apply to Text2SQL and Text2SPARQL, and would justi","A testable extension would be to apply the same five schema formats to a real-world knowledge graph with several hundred node types and relationships, where exact-match pruning is likely to retain too many elements via accidental string overlaps; the paper's own NER-masking and similarity variants would then become the main contenders.","The finding that execution-based accuracy drops by only around two percent for two of three models when schemas are pruned suggests that most of the information an LLM needs is already encoded in the question itself. This strengthens the case for treating schema linking as a recoverable-from-failure step rather than an all-or-nothing prerequisite.","The paper measures cost only through input tokens. A more complete cost model would include the latency of the pruning step itself, the cost of misgenerated queries that must be repaired by a human, and the cost of serving output tokens; these factors could alter the threshold at which pruning stops being beneficial."],"forward_implications":["For applications built on moderately sized models, serving a Text2Cypher system with per-question exact-match schema pruning should be the default baseline rather than the full-schema prompt, because it simultaneously improves measured accuracy and reduces input tokens.","For very large models with long context windows, pruning will not necessarily improve accuracy, but it can still cut costs enough to be worthwhile in high-volume serving.","The 'NER Masked' variant demonstrates a concrete failure mode of naive exact matching: string overlap between a named entity and a schema element can retain irrelevant properties. Applying entity-type masking before matching is a cheap, testable fix that may matter more on domains with many ambiguous labels.","Because the paper's heuristic pruning operates on node labels, relationship types, and properties rather than on query intent, its effectiveness may depend on how well the natural-language question names database elements verbatim; richer languages that paraphrase schema terms will likely need similarity-based or LLM-assisted filtering.","The cost calculations, which scale linearly with input token count and hold output length constant, offer a practical planning equation for anyone choosing between self-hosted and API-based LLM deployment for graph query generation."],"supporting_citations":[{"why":"Supplies the Text2Cypher dataset, the prompt template, and the benchmark instances used for training and testing.","marker":"[18]"},{"why":"Motivates schema linking for query generation and provides a prior Text2SQL method; used to contextualize the finding that schema filtering helps smaller models.","marker":"[2]"},{"why":"Provides the prior observation that LLMs with extended context windows need schema filtering less, which the paper reproduces for Gemini-1.5-Flash.","marker":"[3]"},{"why":"Introduces the entity-to-schema dictionary mapping and A* shortest-path approach that exact-match schema filtering builds on.","marker":"[23]"},{"why":"Presents a preceding Text2GQL framework that also uses schema extraction and LLM-based filtering for natural-language-to-graph-query tasks.","marker":"[36]"},{"why":"The basis for the paper's caveat that excessive filtering can remove essential schema components and harm accuracy.","marker":"[26]"},{"why":"An early Text2SQL schema-linking work using string matching, cited as the antecedent of the paper's heuristic approach.","marker":"[5]"},{"why":"A second early Text2SQL heuristic schema-linking method, cited alongside IRNet as precedent for string-match-based filtering.","marker":"[27]"}],"fun_headline_variants":["Schema filtering slashes token counts and boosts Text2Cypher accuracy","Pruned schemas make Text2Cypher cheaper and sharper for small LLMs","Exact-match schema pruning cuts prompt size, lifts query accuracy","Token-trimming schema filters boost Text2Cypher accuracy and cut costs","Dynamic schema pruning trims tokens, enhances Text2Cypher for smaller LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark databases are demo-oriented and simple, with the longest schema around 2,700 tokens, so the finding that exact-match schema pruning helps may not hold for real-world graph schemas that are much larger, much more deeply nested, or full of overlapping labels.","fun_headline_variants_meta":{"raw":{"variants":["Schema filtering slashes token counts and boosts Text2Cypher accuracy","Pruned schemas make Text2Cypher cheaper and sharper for small LLMs","Exact-match schema pruning cuts prompt size, lifts query accuracy","Token-trimming schema filters boost Text2Cypher accuracy and cut costs","Dynamic schema pruning trims tokens, enhances Text2Cypher for smaller LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1576,"prompt_tokens":860,"completion_tokens":716,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":614}},"tokens_in":476,"tokens_out":716,"duration_ms":6579,"temperature":1.0,"reasoning_tokens":614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:11:52.041622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real-world knowledge graph with thousands of node labels and relationship types, build a test set with naturally phrased questions that do not literally repeat schema element names, and measure execution-based exact-match accuracy for Llama-3.1-8B under the five schema formats. If exact-match pruning performs worse than the full Base Schema on that test set, the paper's central claim would fail for the regime it is meant to generalize to.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Text2Cypher dataset, the prompt template, and the benchmark instances used for training and testing."},{"cited_title":"Liang, K","cited_arxiv_id":null,"evidence_quote":"Introduces the entity-to-schema dictionary mapping and A* shortest-path approach that exact-match schema filtering builds on."}],"review_version":1}