Pith. sign in

REVIEW 3 major objections 4 minor 17 references

EICopilot: Search and Explore Enterprise Information over Large-scale Knowledge Graphs with LLM-driven Agents

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Masking entity names in both user queries and stored examples lets an LLM-driven agent retrieve intent-matched demonstrations, cutting Gremlin script syntax errors to as low as 10% and raising execution correctness to over 82% on a…

desk verdict The deployed system and masking idea are worth a look, but Table 3's headline numbers don't survive contact with the stated test-set size: the best rows are computed on 56 queries, not 150. read the letter →

arxiv 2501.13746 v1 pith:TJF2CTF6 submitted 2025-01-23 cs.IR cs.AI

Yuhui Yun , Huilong Ye , Xinru Li , Ruojia Li , Jingfeng Deng , Li Li , Haoyi Xiong This is my paper · ORCID
classification cs.IRcs.AI
keywords LargeLanguageModelsKnowledgeGraphsGremlinIn-contextLearningQueryMaskingEnterpriseSearchRetrieval-AugmentedGenerationText-to-Graph
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a natural-language chatbot can reliably query a very large enterprise knowledge graph when it is built as an agent that retrieves few-shot examples of similar query intent from a curated vector database and then feeds them to an LLM that writes Gremlin scripts. The key move is masking entity names in both the user question and the stored examples before similarity matching, so that retrieval aligns with intent rather than with company names. On a test set of 150 real user queries, the Full Mask variant lowers the syntax error rate for Gremlin scripts to as low as 2.00% for a fine-tuned model and 10.00% for a pretrained model, and raises execution correctness to as high as 82.14–83.93%, compared with zero-shot baselines that err 15–53% of the time and score only 17–41% on correctness. The gains hold across several LLM backbones, including an open 8-billion-parameter model.

What carries the argument

The load-bearing object is the vector database of representative query–Gremlin pairs, built offline by collecting real user queries, having experts write Gremlin scripts, and augmenting the seeds with Graph2NL and feedback-based regeneration. The central mechanism is the query masking strategy: before embedding and cosine similarity, both the evaluation query and each stored representative query have recognized entity names (company names, personal names) replaced by a placeholder token, so retrieval returns examples that match the structural intent of the question. The selected top-k pairs are then injected into the LLM prompt alongside schema metadata through a two-stage schema-linking step, and a reflection module checks and corrects edges, directions, attributes, and syntax before execution.

What would settle it

Take a fixed corpus of 200 annotated queries, build the vector database for 100 of them, and test on the other 100 that contain mostly unseen intent clusters; if Full Mask matches zero-shot accuracy on those held-out intents, the reported gains are driven by seed-set coverage of the test queries rather than by masked intent retrieval.

Watch

Extended reading notes

Core claim

On a dataset of 150 queries drawn from real user traffic and annotated with the intended Gremlin script, EICopilot with Full Mask consistently outperforms both zero-shot generation and the same agent with alternative matching strategies. When both the evaluation query and the representative queries have their company entities replaced by a placeholder before vector retrieval, the top-5 retrieved examples are questions with the same underlying intent (e.g., 'who are the executives') rather than questions about the same company; feeding those intent-matched examples to the LLM as in-context demonstrations yields syntax error rates between 2.00% and 30.67% across models and execution correctness scores reaching as high as 83.93%. The paper argues that entity masking removes the dominant surface feature that causes raw vector matching to retrieve same-entity/different-intent examples, and that the improvement holds for both proprietary and open-weights LLMs.

Load-bearing premise

The masking trick only helps if a question's intent is already represented by at least one example in the vector database; if no stored question has the same intent, retrieved demonstrations will not supply the right Gremlin pattern, and the benefit of masking disappears.

Editorial extensions

If this is right

  • If the reported effectiveness of Full Mask holds, enterprise information search can be reduced to a single conversational turn: a user asks a question in natural language and receives a summarised answer backed by an executing Gremlin query, replacing the current multi-step manual subgraph exploration.
  • Because the vector database is constructed once offline and the online component only retrieves demonstrations and prompts an LLM, the same architecture can be applied to a new knowledge graph by rebuilding the seed set, without retraining the underlying LLM.
  • The masking strategy decouples intent from entity identity, suggesting the technique can be transferred to other structured query generation tasks—Cypher, SPARQL, or text-to-SQL—where example selection by raw embedding similarity is distorted by dominant named entities.
  • The production deployment with thousands of daily users indicates that agentic LLM workflows over billion-scale graphs are stable enough for real-world use, not only for offline benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Masking may also help when the entity vocabulary is large but the intent space is small: a test that takes a fixed set of intents and varies the entities inside them would show whether the 2–10% syntax error floor comes from residual intent confusion rather than from unmasked surface overlap.
  • The paper does not report a cold-start variant with a brand-new graph schema and no pre-built seed set; if the vector database is empty or sparse, the agent would fall back to zero-shot behavior, so the true boundary of the method lies in how many curated examples are needed per intent cluster.
  • One could measure the contribution of the reflection module separately from masking by running Full Mask with reflection disabled; the ablations in the paper only vary the matching strategy, not the later correction step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces EICopilot, an LLM-driven agent system for natural-language search and exploration over a large-scale enterprise knowledge graph. The system uses a vector database of representative query?Gremlin pairs for in-context learning, a Chain-of-Thought reasoning pipeline, and a query-masking strategy that strips entity names from both the user query and candidate examples before similarity matching. The authors report that the Full Mask variant reduces the Gremlin syntax error rate to as low as 10.00% and achieves execution correctness up to 82.14%, outperforming zero-shot baselines and other masking variants. The evaluation is based on 150 real-world queries collected from Baidu Enterprise Search, with expert scoring of execution correctness, and the system is reported as deployed with 5000+ daily active users.

Significance. If the reported results are valid, EICopilot is a useful industrial-strength demonstration of LLM agents for knowledge-graph query generation, and the query-masking strategy offers a simple, plausible mechanism for improving intent-based retrieval of ICL examples. The paper is honest about its scope, explicitly noting that the contribution is the system design and agentic workflow rather than the underlying LLMs. The deployment details and the case study help ground the work. However, the current evaluation has serious reporting inconsistencies that prevent the central quantitative claims from being accepted as stated.

major comments (3)
  1. [Section 3.2 / Table 3] The execution-correctness values for the top-5 Rep. Mask and Full Mask rows are not representable as sums of 0.5-point scores out of the stated 150-query test set. For example, 82.14% corresponds exactly to 46/56, and 83.93% corresponds to 47/56; each of the other values in those rows also has denominator 56 (e.g., 76.79% = 43/56). The paper states in Section 3.1.1 that the test dataset consists of 150 entries and in Section 3.2 that 150 queries from real-world traffic were used, with no mention of any 56-query subset. As written, the headline Full Mask and Rep. Mask numbers are computed on a different, unreported sample than the zero-shot and Raw Match baselines, making the head-to-head comparison invalid and undermining the abstract's claim of execution correctness up to 82.14% and the conclusion's 83.93%.
  2. [Section 3.1.2] The Syntax Error Rate metric is defined as 'the percentage of predicted Gremlin scripts that are free of syntactic errors,' but the formula given is 1 - (1/N) * sum of indicator(execution success), where the indicator is 1 if execution succeeds and 0 if execution fails. This formula measures the execution-failure rate, not the syntactic-error rate. A script with valid syntax can fail at runtime (e.g., due to a timeout, a type error, or a missing property), and such failures would be incorrectly counted as syntactic errors. Conversely, the interpretation of the reported numbers in Section 3.2 as syntax error rates is therefore misleading. The authors should either rename the metric to 'execution failure rate' or, if syntactic validity is what they intend, replace the formula with a check that parses or validates the script syntax independently of execution.
  3. [Section 3.1.4] The descriptions of the Rep. Mask and Eval. Mask strategies are identical: both say 'the entities in each representative query are masked before performing vector-based matching.' Given the strategy names, one of them should refer to masking the evaluating query rather than the representative queries. This is a clear typographical/definitional error that makes the ablation comparison hard to interpret.
minor comments (4)
  1. [Section 3.1.1] The complexity-score thresholds in Equations (1) and (2) are stated without justification, and the assignment of points to operators in Table 1 (e.g., groupCount and by each contributing 2 points) appears arbitrary. Since this score is only used to characterize the dataset, a brief justification or a citation would help.
  2. [Table 3] The zero-shot Llama row reports 17.66% execution correctness. With the stated 0/0.5/1 scoring and a denominator of 150, the only representable values near this are 17.33% (26/150) and 17.67% (26.5/150). This looks like a typo and should be corrected.
  3. [Appendix A] The deployment appendix refers to NebulaGraph as the graph database and then mentions 'The HugeGraph system comprises hundreds of millions of nodes.' Earlier sections describe the system as built on Apache TinkerPop. The relationship between NebulaGraph, HugeGraph, and TinkerPop is never clarified, which is confusing for a reader trying to understand the deployment.
  4. [General] There are numerous typos and grammatical errors, including 'an novel' in the abstract, 'propsoed' in the introduction, 'alike querues' in Section 3.4, and inconsistent capitalization. A thorough proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical comparisons evaluated on an independent, non-overlapping test set, not derivations from fitted inputs or self-citations.

full rationale

EICopilot is an empirical systems paper. The claimed results (syntax error rate down to 10.00% and execution correctness up to 82.14% / 83.93%) are measured outcomes of a pipeline on a 150-query real-traffic test set, with the test set stated to have no overlap with SFT training and validation data (Section 3.2: 'there is no overlap between the testing dataset for evaluation and the training/validation datasets for SFT'). The ICL representative queries come from a separately constructed seed repository, and the four matching strategies are evaluated explicitly; Full Mask is not fitted to the test set, and no parameter is tuned on the evaluation queries. The paper contains no derivation that defines a predicted quantity in terms of an input, no fitted parameter renamed as a prediction, and no load-bearing self-citation chain; references to prior LLM and RAG work are contextual. Two internal inconsistencies exist—the 'Syntax Error Rate' formula in Section 3.1.2 uses execution success rather than syntactic validity, and the top-5 Rep. Mask / Full Mask execution-correctness values in Table 3 are not representable with the stated denominator of 150—but these are measurement and commensurability flaws, not circularity: they do not make any result true by construction. The pre-assessed circularity concern (score 1.0) does not rise to a circularity finding; the reviewable issues belong in a correctness review, not a circularity pass.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its claims rest on modeling assumptions about LLM capabilities and the effectiveness of entity masking for intent matching, plus configurable scoring thresholds for dataset characterization.

free parameters (3)
  • Operator complexity scores = has=1, out=1, groupCount=2, repeat=3 (Table 1)
    Hand-assigned weights used only to classify test queries into difficulty levels in Section 3.1.1, not used in the reported accuracy metrics.
  • Traversal length score thresholds = 1 if steps<5, 2 if 5-7, 3 otherwise
    Arbitrary thresholds in Equation (1) used to assign difficulty levels to test queries.
  • Number of ICL examples (top-3 vs top-5) = 3 or 5
    Configuration choice in Section 3.1.4; the paper reports results for both without a principled selection criterion.
assumptions (3)
  • domain assumption LLMs with ICL can generate syntactically valid and semantically correct Gremlin scripts when provided with a few relevant examples.
    The entire system depends on this. It is tested only on three model families and 150 queries, so it may not hold for other LLMs or query types.
  • domain assumption Masking entity names in queries preserves the intent for similarity-based example matching.
    This is the paper's novel strategy (Sections 2.1.3, 2.2.4, 3.1.4). The paper provides only empirical evidence on one dataset, no theoretical justification.
  • domain assumption The 150-query test set is representative of real user traffic for enterprise information search.
    The queries are sampled from Aiqicha real traffic, but no distribution analysis, demographic information, or public data is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EICopilot: Search and Explore Enterprise Information over Large-scale Knowledge Graphs with LLM-driven Agents." pith.science (2026). https://pith.science/paper/TJF2CTF6

@misc{pith2026250113746,
  author       = {Pith},
  title        = {Pith review of: EICopilot: Search and Explore Enterprise Information over Large-scale Knowledge Graphs with LLM-driven Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TJF2CTF6}},
  note         = {Machine review of arXiv:2501.13746}
}
read the original abstract

The paper introduces EICopilot, an novel agent-based solution enhancing search and exploration of enterprise registration data within extensive online knowledge graphs like those detailing legal entities, registered capital, and major shareholders. Traditional methods necessitate text-based queries and manual subgraph explorations, often resulting in time-consuming processes. EICopilot, deployed as a chatbot via Baidu Enterprise Search, improves this landscape by utilizing Large Language Models (LLMs) to interpret natural language queries. This solution automatically generates and executes Gremlin scripts, providing efficient summaries of complex enterprise relationships. Distinct feature a data pre-processing pipeline that compiles and annotates representative queries into a vector database of examples for In-context learning (ICL), a comprehensive reasoning pipeline combining Chain-of-Thought with ICL to enhance Gremlin script generation for knowledge graph search and exploration, and a novel query masking strategy that improves intent recognition for heightened script accuracy. Empirical evaluations demonstrate the superior performance of EICopilot, including speed and accuracy, over baseline methods, with the \emph{Full Mask} variant achieving a syntax error rate reduction to as low as 10.00% and an execution correctness of up to 82.14%. These components collectively contribute to superior querying capabilities and summarization of intricate datasets, positioning EICopilot as a groundbreaking tool in the exploration and exploitation of large-scale knowledge graphs for enterprise information search.

Figures

Figures reproduced from arXiv: 2501.13746 by the authors.

Figure 1
Figure 1. Enterprise Information Search and Exploration: (a) conducting a manual search of the enterprise, (b) manually researching the major shareholders of the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The Overall framework of EICopilot. and scripts) with similar intent, such as those regarding a com￾pany’s financial status or actual controller, which share similar syntax or logic in scripts. our work finds that masking entity names in queries enhances accuracy in intent matching. In this way, we propose a novel query masking strategy to improve ICL example matching and accurately interpret user intent, enhancing … view at source ↗
Figure 3
Figure 3. Schema linking prompt. 2.2.3 Schema Linking. In practical commercial environments, the challenge of managing large graph databases is often constrained by the context bandwidth of LLM prompts. This contextual limitation renders it impractical to integrate the comprehensive architectural details of all graph data tables into a single prompt. Moreover, incorporating excessive schema information could adversely affect … view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Schema information prompt. 2.2.4 Query Language Generation. The module aims to accurately deconstruct user queries by obfuscating key information. The core process involves separating the user’s query content from the in￾tent and then leveraging a knowledge base in a v…
Figure 5
Figure 5. Figure 5: The difficulty of Real Traffic Dataset [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Top-5 Matched Representative Queries using Raw matching [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Top-5 Matched Representative Queries using Rep. Mask [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Graph database deployment architecture B EXAMPLES OF PROMPTS AND OUTPUTS B.1 Prompt: Gremlin Generation. The Gremlin generation prompt, as shown in [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Gremlin generation prompt C SCRIPTS OF TOP MATCHED REPRESENTATIVE QUERIES We provide the Top-5 representative query-Corresponding gremlin script pairs retrieved from the vector database, as shown in [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 4 canonical work pages

  1. [1]

    Ion Androutsopoulos, Graeme D Ritchie, and Peter Thanisch. 1995. Natural language interfaces to databases–an introduction. Natural language engineering 1, 1 (1995), 29–81

    1995

  2. [2]

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Ruther- ford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bog- dan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning . PMLR, 2206–2240

    2022

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  4. [4]

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023)

    arXiv 2023

  5. [5]

    Naihao Deng, Yulong Chen, and Yue Zhang. 2022. Recent Advances in Text-to- SQL: A Survey of What We Have and What We Expect. InProceedings of the 29th International Conference on Computational Linguistics (COLING) . International Committee on Computational Linguistics, 2166–2187

    work page 2022

  6. [6]

    Yujuan Ding, Wenqi Fan, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meets llms: Towards retrieval- augmented large language models. arXiv preprint arXiv:2405.06211 (2024)

    arXiv 2024

  7. [7]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)

    arXiv 2023

  8. [8]

    Tong Guo and Huilin Gao. 2019. Content enhanced bert-based text-to-sql gener- ation. arXiv preprint arXiv:1910.07179 (2019)

    work page Pith review arXiv 2019

Show all 17 references
  1. [9]

    Pengjun Li, Qixin Zhao, Yingmin Liu, Chao Zhong, Jinlong Wang, and Zhihan Lyu. 2024. Survey and Prospect for Applying Knowledge Graph in Enterprise Risk Management. Computers, Materials & Continua 78, 3 (2024)

    2024

  2. [10]

    Alejandro Lozano, Scott L Fleming, Chia-Chun Chiang, and Nigam Shah. 2023. Clinfo. ai: An open-source retrieval-augmented large language model system for answering medical questions using scientific literature. In PACIFIC SYMPOSIUM ON BIOCOMPUTING 2024 . World Scientific, 8–23

    2023

  3. [11]

    Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu

  4. [12]

    Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christo- pher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. arXiv preprint arXiv:2201.11227 (2022)

    2022 arXiv

  5. [13]

    Yunqi Qiu, Kun Zhang, Yuanzhuo Wang, Xiaolong Jin, Long Bai, Saiping Guan, and Xueqi Cheng. 2020. Hierarchical query graph generation for complex ques- tion answering over knowledge graph. In Proceedings of the 29th ACM interna- tional conference on information & knowledge man...

    2020

  6. [14]

    Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2019. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv:1911.04942 (2019)

    2019 arXiv

  7. [15]

    Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. 2024. Give us the Facts: Enhancing Large Language Models With Knowledge Graphs for Fact- Aware Language Modeling. IEEE Transactions on Knowledge and Data Engineering 36, 7 (2024), 3091–3110. https://doi.org/10.110...

    2024

  8. [16]

    What is the official website of World Kitchen (Shanghai) Co., Ltd.?

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chen- long Deng, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107 (2023). A DEPLOYMENT The deployment of our system, illustrated in F...

    2023

  9. [2024]

    IEEE Transactions on Knowledge and Data Engineering 36, 7 (2024), 3580–3599

    Unifying Large Language Models and Knowledge Graphs: A Roadmap. IEEE Transactions on Knowledge and Data Engineering 36, 7 (2024), 3580–3599. https://doi.org/10.1109/TKDE.2024.3352100

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.