{"id":"67bb72a5-fa75-43da-9ca1-60ea3d4c01ae","arxiv_id":"2501.13746","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EICopilot uses LLM agents with in-context learning and entity masking to generate Gremlin queries for enterprise knowledge graph search, reporting better accuracy than zero-shot baselines on 150 internal queries.","lead":"This paper describes EICopilot, a chatbot that converts natural language questions about Chinese companies into Gremlin queries over a large enterprise knowledge graph. It reports that a query-masking trick for picking in-context examples lowers syntax errors and improves execution correctness compared with zero-shot LLM baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's top-5 Rep. Mask and Full Mask execution-correctness values are consistent only with an undocumented 56-query subset, not the stated 150-query test set, so the headline 82.14% claim is not commensurable with the baselines.","rationale":"We agree with the reader that the paper describes a deployed, plausible system, and the masking strategy is an interesting contribution. However, the single most load-bearing concern is not the masking assumption but the internal inconsistency in the evaluation metrics. Section 3.1.2 defines 'Syntax Error Rate' as the percentage of scripts free of syntactic errors, then computes it as 1 minus the execution-success rate; this conflates syntactic validity with execution success, so the abstract's '10.00% syntax error rate' is really an execution-failure rate. More decisively, the execution-correctness numbers in Table 3 are numerically impossible under the stated protocol. The test set has 150 queries and scores are 0, 0.5, or 1, so any percentage must be a multiple of 1/150 = 0.6667% (or 0.3333% if 0.5 scores are allowed). The zero-shot and top-3 rows, and the top-5 Raw Match and Eval. Mask rows, all satisfy this (e.g., 41.00% = 61.5/150). But the top-5 Rep. Mask and Full Mask rows (76.79%, 61.61%, 81.25%, 65.18%, 74.11%, and 82.14%, 65.18%, 83.93%, 69.64%, 79.46%) do not: they are exactly representable with denominator 56 (e.g., 46/56 = 82.14%). The paper never states that a 56-query subset was used for these conditions, so the superior results for the two best-performing strategies are based on a different, unexplained sample. This directly undermines the headline claim that EICopilot achieves 82.14% execution correctness, because the comparison against the 150-query baselines is not apples-to-apples. This is not a matter of external consensus or generalization; it is an internal inconsistency that can be settled by inspecting the raw scores. We therefore recommend the paper be marked UNVERDICTED until the authors provide the full per-query evaluation data and recompute the tables with a consistent denominator.","tokens_in":13598,"tokens_out":10082,"duration_ms":75422,"concrete_test":"Independently recompute the Table 3 percentages from the per-query expert scores, using the stated 150-query test set and the 0/0.5/1 scoring rule for every condition; if the top-5 Rep. Mask and Full Mask rows cannot be reproduced at denominator 150, the reported advantages are artifacts of an inconsistent evaluation subset.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central quantitative claim rests on Tables 2 and 3. In Section 3.1.2, 'Syntax Error Rate' is defined as the percentage of scripts free of syntactic errors, but the formula is 1 − (1/N)Σ𝟙(execution success), so the reported 'syntax error rate' is actually an execution-failure rate; a script that times out or throws a runtime error is counted as a 'syntax error.' More seriously, the execution-correctness percentages in Table 3 are internally inconsistent with the stated 150-query test set and the 0/0.5/1 scoring rule. While most rows (zero-shot, top-3, and top-5 Raw Match / Eval. Mask) are representable as sums of 0.5-point scores out of 150 (e.g., 41.00% = 61.5/150), the top-5 Rep. Mask and Full Mask rows (76.79%, 61.61%, 81.25%, 65.18%, 74.11% and 82.14%, 65.18%, 83.93%, 69.64%, 79.46%) are not representable with denominator 150; the exact values require a denominator of 56 (e.g., 46/56 = 82.14%). The paper never mentions a 56-query subset, so the superior numbers for the two masking strategies appear to be computed on a different, smaller sample than the baselines, making the head-to-head comparison invalid. This is a load-bearing flaw because the abstract's headline 'execution correctness of up to 82.14%' and the conclusion's '83.93%' are drawn from these rows.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EICopilot, an LLM-driven agent system for natural-language search and exploration over a large-scale enterprise knowledge graph. The system uses a vector database of representative query?Gremlin pairs for in-context learning, a Chain-of-Thought reasoning pipeline, and a query-masking strategy that strips entity names from both the user query and candidate examples before similarity matching. The authors report that the Full Mask variant reduces the Gremlin syntax error rate to as low as 10.00% and achieves execution correctness up to 82.14%, outperforming zero-shot baselines and other masking variants. The evaluation is based on 150 real-world queries collected from Baidu Enterprise Search, with expert scoring of execution correctness, and the system is reported as deployed with 5000+ daily active users.","tokens_in":13983,"tokens_out":3195,"duration_ms":26849,"significance":"If the reported results are valid, EICopilot is a useful industrial-strength demonstration of LLM agents for knowledge-graph query generation, and the query-masking strategy offers a simple, plausible mechanism for improving intent-based retrieval of ICL examples. The paper is honest about its scope, explicitly noting that the contribution is the system design and agentic workflow rather than the underlying LLMs. The deployment details and the case study help ground the work. However, the current evaluation has serious reporting inconsistencies that prevent the central quantitative claims from being accepted as stated.","major_comments":[{"comment":"The execution-correctness values for the top-5 Rep. Mask and Full Mask rows are not representable as sums of 0.5-point scores out of the stated 150-query test set. For example, 82.14% corresponds exactly to 46/56, and 83.93% corresponds to 47/56; each of the other values in those rows also has denominator 56 (e.g., 76.79% = 43/56). The paper states in Section 3.1.1 that the test dataset consists of 150 entries and in Section 3.2 that 150 queries from real-world traffic were used, with no mention of any 56-query subset. As written, the headline Full Mask and Rep. Mask numbers are computed on a different, unreported sample than the zero-shot and Raw Match baselines, making the head-to-head comparison invalid and undermining the abstract's claim of execution correctness up to 82.14% and the conclusion's 83.93%.","section":"Section 3.2 / Table 3"},{"comment":"The Syntax Error Rate metric is defined as 'the percentage of predicted Gremlin scripts that are free of syntactic errors,' but the formula given is 1 - (1/N) * sum of indicator(execution success), where the indicator is 1 if execution succeeds and 0 if execution fails. This formula measures the execution-failure rate, not the syntactic-error rate. A script with valid syntax can fail at runtime (e.g., due to a timeout, a type error, or a missing property), and such failures would be incorrectly counted as syntactic errors. Conversely, the interpretation of the reported numbers in Section 3.2 as syntax error rates is therefore misleading. The authors should either rename the metric to 'execution failure rate' or, if syntactic validity is what they intend, replace the formula with a check that parses or validates the script syntax independently of execution.","section":"Section 3.1.2"},{"comment":"The descriptions of the Rep. Mask and Eval. Mask strategies are identical: both say 'the entities in each representative query are masked before performing vector-based matching.' Given the strategy names, one of them should refer to masking the evaluating query rather than the representative queries. This is a clear typographical/definitional error that makes the ablation comparison hard to interpret.","section":"Section 3.1.4"}],"minor_comments":[{"comment":"The complexity-score thresholds in Equations (1) and (2) are stated without justification, and the assignment of points to operators in Table 1 (e.g., groupCount and by each contributing 2 points) appears arbitrary. Since this score is only used to characterize the dataset, a brief justification or a citation would help.","section":"Section 3.1.1"},{"comment":"The zero-shot Llama row reports 17.66% execution correctness. With the stated 0/0.5/1 scoring and a denominator of 150, the only representable values near this are 17.33% (26/150) and 17.67% (26.5/150). This looks like a typo and should be corrected.","section":"Table 3"},{"comment":"The deployment appendix refers to NebulaGraph as the graph database and then mentions 'The HugeGraph system comprises hundreds of millions of nodes.' Earlier sections describe the system as built on Apache TinkerPop. The relationship between NebulaGraph, HugeGraph, and TinkerPop is never clarified, which is confusing for a reader trying to understand the deployment.","section":"Appendix A"},{"comment":"There are numerous typos and grammatical errors, including 'an novel' in the abstract, 'propsoed' in the introduction, 'alike querues' in Section 3.4, and inconsistent capitalization. A thorough proofreading pass is needed.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The denominator inconsistency in Table 3 is the most serious issue. It would be prudent to ask the authors for the raw per-query expert scores for all configurations, so the editor can verify that the reported percentages correspond to a consistent evaluation protocol. The paper also lacks significance tests or confidence intervals, which is not fatal for a systems paper, but combined with the reporting issues it raises the bar for transparency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zeynep, quick take on arXiv:2501.13746.\n\nThe paper describes EICopilot, an LLM-agent pipeline that turns natural-language questions about Chinese company registries into Gremlin scripts over a large TinkerPop graph. The genuinely new piece is the Full Mask ICL strategy: mask entity names in both the user query and the candidate representative queries, embed, and retrieve examples by masked intent rather than surface lexical match. That's a sensible, modest idea, and the case study (Figures 6-8) shows it changes what the retriever returns. If the numbers were clean, this would be a useful engineering report for anyone building NL-to-GraphQL/Gremlin systems.\n\nThey're not clean. The stress-test caught it, and I checked: Table 3's top-5 Rep. Mask and Full Mask execution-correctness values (76.79%, 82.14%, 81.25%, etc.) cannot be represented as sums of 0.5-point scores out of 150 queries. They line up perfectly with a 56-query denominator (46/56 = 82.14%). The paper never mentions a 56-query subset. So the headline 'up to 82.14%' and the conclusion's '83.93%' are computed on a different, smaller sample than the baselines they're compared against. That makes the central quantitative claim incommensurable as reported.\n\nThen there's the 'Syntax Error Rate' definition, which is 1 − (1/N)Σ𝟙(execution success). That's the execution-failure rate, not a measure of syntax errors. A timeout or runtime exception gets counted as a syntax error. Mislabeled.\n\nThe rest of the evaluation is thin in ways the paper itself half-admits: 150 internal queries, no significance tests, expert scoring that's necessarily subjective, and the only baselines are zero-shot promptings of the same LLMs. No comparison to fine-tuned text-to-SQL systems or alternate ICL samplers. The 'Operator Complexity' scoring is also ad hoc.\n\nWhat's genuinely good: the system is deployed (5,000+ DAU), the architecture is described in enough detail to replicate the approach, and the masking idea is well-motivated. The paper is honest about scope limitations.\n\nBottom line: the idea deserves a serious referee, but the paper cannot be accepted as-is. The authors need to redo Table 3 so all rows use the same test set, fix the metric definition, and ideally add a couple of nontrivial baselines. I'd engage with it, but I'd want the revision before trusting any of the numbers.","headline":"The deployed system and masking idea are worth a look, but Table 3's headline numbers don't survive contact with the stated test-set size: the best rows are computed on 56 queries, not 150.","tokens_in":14488,"tokens_out":4178,"would_cite":true,"duration_ms":33142,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Masking entity names in both user queries and stored examples lets an LLM-driven agent retrieve intent-matched demonstrations, cutting Gremlin script syntax errors to as low as 10% and raising execution correctness to over 82% on a…","keywords":["Large Language Models","Knowledge Graphs","Gremlin","In-context Learning","Query Masking","Enterprise Search","Retrieval-Augmented Generation","Text-to-Graph Query"],"falsifier":"Take a fixed corpus of 200 annotated queries, build the vector database for 100 of them, and test on the other 100 that contain mostly unseen intent clusters; if Full Mask matches zero-shot accuracy on those held-out intents, the reported gains are driven by seed-set coverage of the test queries rather than by masked intent retrieval.","tokens_in":13433,"feed_emoji":"🤖","tokens_out":10669,"duration_ms":82085,"temperature":0.7,"pith_summary":"This paper claims that a natural-language chatbot can reliably query a very large enterprise knowledge graph when it is built as an agent that retrieves few-shot examples of similar query intent from a curated vector database and then feeds them to an LLM that writes Gremlin scripts. The key move is masking entity names in both the user question and the stored examples before similarity matching, so that retrieval aligns with intent rather than with company names. On a test set of 150 real user queries, the Full Mask variant lowers the syntax error rate for Gremlin scripts to as low as 2.00% for a fine-tuned model and 10.00% for a pretrained model, and raises execution correctness to as high as 82.14–83.93%, compared with zero-shot baselines that err 15–53% of the time and score only 17–41% on correctness. The gains hold across several LLM backbones, including an open 8-billion-parameter model.","feed_headline":"Masked-example retrieval cuts Gremlin script errors to 10%","feed_subtitle":"Masking company names keeps the LLM focused on intent, boosting correct Gremlin execution to 82%.","key_machinery":"The load-bearing object is the vector database of representative query–Gremlin pairs, built offline by collecting real user queries, having experts write Gremlin scripts, and augmenting the seeds with Graph2NL and feedback-based regeneration. The central mechanism is the query masking strategy: before embedding and cosine similarity, both the evaluation query and each stored representative query have recognized entity names (company names, personal names) replaced by a placeholder token, so retrieval returns examples that match the structural intent of the question. The selected top-k pairs are then injected into the LLM prompt alongside schema metadata through a two-stage schema-linking step, and a reflection module checks and corrects edges, directions, attributes, and syntax before execution.","core_discovery":"On a dataset of 150 queries drawn from real user traffic and annotated with the intended Gremlin script, EICopilot with Full Mask consistently outperforms both zero-shot generation and the same agent with alternative matching strategies. When both the evaluation query and the representative queries have their company entities replaced by a placeholder before vector retrieval, the top-5 retrieved examples are questions with the same underlying intent (e.g., 'who are the executives') rather than questions about the same company; feeding those intent-matched examples to the LLM as in-context demonstrations yields syntax error rates between 2.00% and 30.67% across models and execution correctness scores reaching as high as 83.93%. The paper argues that entity masking removes the dominant surface feature that causes raw vector matching to retrieve same-entity/different-intent examples, and that the improvement holds for both proprietary and open-weights LLMs.","pith_inferences":["Masking may also help when the entity vocabulary is large but the intent space is small: a test that takes a fixed set of intents and varies the entities inside them would show whether the 2–10% syntax error floor comes from residual intent confusion rather than from unmasked surface overlap.","The paper does not report a cold-start variant with a brand-new graph schema and no pre-built seed set; if the vector database is empty or sparse, the agent would fall back to zero-shot behavior, so the true boundary of the method lies in how many curated examples are needed per intent cluster.","One could measure the contribution of the reflection module separately from masking by running Full Mask with reflection disabled; the ablations in the paper only vary the matching strategy, not the later correction step."],"forward_implications":["If the reported effectiveness of Full Mask holds, enterprise information search can be reduced to a single conversational turn: a user asks a question in natural language and receives a summarised answer backed by an executing Gremlin query, replacing the current multi-step manual subgraph exploration.","Because the vector database is constructed once offline and the online component only retrieves demonstrations and prompts an LLM, the same architecture can be applied to a new knowledge graph by rebuilding the seed set, without retraining the underlying LLM.","The masking strategy decouples intent from entity identity, suggesting the technique can be transferred to other structured query generation tasks—Cypher, SPARQL, or text-to-SQL—where example selection by raw embedding similarity is distorted by dominant named entities.","The production deployment with thousands of daily users indicates that agentic LLM workflows over billion-scale graphs are stable enough for real-world use, not only for offline benchmarks."],"supporting_citations":[{"why":"Defines the classic task of natural-language interfaces to databases, which this paper extends to Gremlin over knowledge graphs.","marker":"[1]"},{"why":"Demonstrates retrieval-augmented language modelling, the core technique behind the vector database and in-context example selection.","marker":"[2]"},{"why":"Introduces in-context learning, the mechanism by which matched representative queries guide the LLM's Gremlin generation.","marker":"[3]"},{"why":"Surveys text-to-SQL methods that frame the related text-to-Gremlin generation problem and motivate schema linking.","marker":"[5]"},{"why":"Surveys retrieval-augmented generation with LLMs, the framework EICopilot combines with domain-specific seed data.","marker":"[6]"},{"why":"Provides a relation-aware schema linking method for text-to-SQL that the schema-linking module adapts for graph schemas.","marker":"[14]"}],"fun_headline_variants":["Masking company names in queries slashes Gremlin script errors","Entity masking boosts Gremlin script accuracy to 82%","Query masking lets LLMs generate correct Gremlin scripts","Masked intent matching reduces Gremlin syntax errors to 10%","Company-name masking boosts Gremlin script correctness to 82%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The masking trick only helps if a question's intent is already represented by at least one example in the vector database; if no stored question has the same intent, retrieved demonstrations will not supply the right Gremlin pattern, and the benefit of masking disappears.","fun_headline_variants_meta":{"raw":{"variants":["Masking company names in queries slashes Gremlin script errors","Entity masking boosts Gremlin script accuracy to 82%","Query masking lets LLMs generate correct Gremlin scripts","Masked intent matching reduces Gremlin syntax errors to 10%","Company-name masking boosts Gremlin script correctness to 82%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000474,"raw_usage":{"total_tokens":2374,"prompt_tokens":983,"completion_tokens":1391,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1302}},"tokens_in":599,"tokens_out":1391,"duration_ms":9230,"temperature":1.0,"reasoning_tokens":1302,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:37:44.419259+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed corpus of 200 annotated queries, build the vector database for 100 of them, and test on the other 100 that contain mostly unseen intent clusters; if Full Mask matches zero-shot accuracy on those held-out intents, the reported gains are driven by seed-set coverage of the test queries rather than by masked intent retrieval.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys text-to-SQL methods that frame the related text-to-Gremlin generation problem and motivate schema linking."}],"review_version":1}