REVIEW 8 cited by
Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Integrating Large Language Models (LLMs) with Knowledge Graphs (KGs) results in complex systems with numerous hyperparameters that directly affect performance. While such systems are increasingly common in retrieval-augmented generation, the role of systematic hyperparameter optimization remains underexplored. In this paper, we study this problem in the context of Cognee, a modular framework for end-to-end KG construction and retrieval. Using three multi-hop QA benchmarks (HotPotQA, TwoWikiMultiHop, and MuSiQue) we optimize parameters related to chunking, graph construction, retrieval, and prompting. Each configuration is scored using established metrics (exact match, F1, and DeepEval's LLM-based correctness metric). Our results demonstrate that meaningful gains can be achieved through targeted tuning. While the gains are consistent, they are not uniform, with performance varying across datasets and metrics. This variability highlights both the value of tuning and the limitations of standard evaluation measures. While demonstrating the immediate potential of hyperparameter tuning, we argue that future progress will depend not only on architectural advances but also on clearer frameworks for optimization and evaluation in complex, modular systems.
Forward citations
Cited by 8 Pith papers
-
Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data
Setoka evaluates memory-augmented agents on four levels of user understanding—semantic memory, episodic memory, behavior patterns, and personality traits—over synthesized heterogeneous user data, and finds performance...
-
MIRIX: Multi-Agent Memory System for LLM-Based Agents
MIRIX introduces a modular multi-agent architecture with Core, Episodic, Semantic, Procedural, Resource, and Knowledge Vault memories that outperforms RAG baselines by 35% on ScreenshotVQA and reaches 85.4% on LOCOMO.
-
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
MemoryAgentBench is a new multi-turn benchmark assessing four memory competencies in LLM agents—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—showing that existing methods ...
-
Know It, Act on It: Investigating Memory Utilization in LLM Personalization
LLM agents often pass a direct recall question about a user's preference yet fail to act on the same preference in a realistic request — a Know–Act gap that persists even in the best systems and is widest, on average,...
-
Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data
Existing agent memory systems score well on single-fact recall but fall near chance on behavior-pattern and personality-trait inference over heterogeneous synthetic user data.
-
Are We Ready For An Agent-Native Memory System?
A four-module framework is used to benchmark 12 agent memory systems, showing no architecture dominates and that workload alignment plus localized maintenance drive performance and cost.
-
Cost and Accuracy of Long-Term Memory in Distributed Multi-Agent Systems Based on Large Language Models
A two-framework testbed comparison claims mem0 is Pareto-optimal over Graphiti for distributed LLM agents because its lower cost is paired with accuracy that is not significantly different.
-
Improving Language Agents through BREW: Bootstrapping expeRientially-learned Environmental knoWledge
An LLM agent can improve by storing and Monte-Carlo-refining concept-level natural-language recipes mined from its own past trajectories.
Discussion (0). Sign in to comment.