{"id":"c430445b-dadb-4f02-a67a-be88a4d4c590","arxiv_id":"2502.18479","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A generative-AI research assistant connects document chat, multimodal chat, and report generation through shared knowledge bases, with an evaluation claiming hybrid vector and knowledge-graph retrieval outperforms either index alone.","lead":"ScienceSage is a web app that builds a shared knowledge base from documents, images, audio and video, then answers questions with short AI retrieval responses or long generated research reports. Its authors report that combining vector and knowledge-graph retrieval gives higher accuracy and fewer hallucinations than either alone, and the app has been used internally at a consumer goods company.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The faithfulness metric in Eq. (1) is inflated for the custom index because its retrieved context is the union of vector and KG retrieval; the headline superiority may be a metric artifact rather than a real retrieval gain.","rationale":"The reader's weakest assumption focuses on GPT-4 judge bias and dataset validity. That is a legitimate concern, but the most load-bearing issue is more direct and internal: the faithfulness metric defined in Eq. (1) is computed against the retrieved context c(q), and the custom index is described as concatenating vector and KG retrieval results before generating an answer. This gives the custom index a systematically larger c(q), which mechanically raises |V| in the numerator of Eq. (1). The superiority claim on faithfulness, and plausibly on the other metrics, is therefore confounded with context size unless the comparison is controlled. This does not change the overall verdict: the paper is an honest system report whose headline empirical claim needs stronger evidence. The reader already asked for numerical results, statistical tests, and released evaluation artifacts; I would add the requirement that retrieved context size be equalized or otherwise controlled. The proposed concrete test settles the concern directly. If the custom index retains its advantage after context-size equalization, the original claim survives; if not, the paper's central empirical assertion is weakened to 'more context helps,' which is not the same as a superior hybrid retrieval architecture.","tokens_in":9746,"tokens_out":3860,"duration_ms":39520,"concrete_test":"Re-run the RAG evaluation with the total retrieved context size held fixed across the three index types: for each query, let the vector-only and KG-only RAGs retrieve k nodes, and let the custom RAG retrieve a total of k nodes (e.g., k/2 from each index) so that the context passed to the generator has the same size. Report all three metrics before and after this equalization, with numeric values and standard deviations. If the custom index's advantage on faithfulness (and ideally correctness and relevance) shrinks or disappears, the headline claim is an artifact of Eq. (1)'s dependence on context size rather than a genuine retrieval improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that the custom (vector+KG) RAG is superior on correctness, relevance, and faithfulness rests on the evaluation in 'RAG evaluation' and its metrics. There is a structural confound in the faithfulness metric. Eq. (1) defines F = |V|/|S|, where V is the subset of statements in the generated answer that can be inferred from the retrieved context c(q). The paper states that for the custom RAG, 'it first retrieves information from vector index and KG index, separately. It then combines all retrieved information together' before synthesizing an answer. Thus c(q) for the custom index is the union of two retrieved contexts, while the vector-only and KG-only RAGs each use a single context. Because adding retrieved nodes can only add verifiable statements (or leave them unchanged), F is monotonically nondecreasing in the size of c(q). The custom index therefore has a mechanical advantage on the faithfulness/hallucination metric that is independent of retrieval quality. The same union effect can also inflate correctness and relevance, because the generator sees more supporting evidence, and the GPT-4 judge may reward longer, more detailed answers. The paper reports only bar charts (Fig. 3, A1–A7) with no numeric values or statistical tests, so the claimed 'consistent' superiority cannot be separated from this context-size confound. The LLM-judge reliability concern identified by the reader is real, but the metric's dependence on context size is a more direct, internal threat to the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ScienceSage, an MVP web application that lets researchers build, store, update, and query knowledge bases encoded as both vector and knowledge-graph indices. The system supports three functions: generating research reports from the internet, chatting with uploaded documents via a RAG system, and chatting with multimodal data. The central evaluation compares three RAG variants (vector-only, KG-only, and a custom vector+KG hybrid) on a dataset of 2,295 queries derived from Smith et al. (2008), using GPT-4-based correctness, relevance, and faithfulness metrics. The paper claims that the custom hybrid index consistently achieves the highest correctness, relevance, and faithfulness across easy, medium, and hard queries, with only a small compromise in speed.","tokens_in":10008,"tokens_out":2632,"duration_ms":28071,"significance":"If the evaluation were sound, the paper would provide a useful practical demonstration that a hybrid vector-plus-knowledge-graph RAG can outperform either index alone on a research-oriented question-answering task. The system is deployed and used in an industrial R&D setting, and the authors share the code and are transparent about the MVP nature of the work. However, the headline empirical claim rests entirely on an evaluation that has a structural confound in the faithfulness metric and lacks the statistical detail needed to support the stated conclusions. The paper is therefore a reasonable system description, but its central performance claim needs substantially stronger evidence before it can be accepted.","major_comments":[{"comment":"The faithfulness metric F = |V|/|S| is structurally confounded with the size of the retrieved context c(q). The custom RAG is described as first retrieving from the vector index and KG index separately and then combining all retrieved information, so its c(q) is the union of two contexts, whereas the vector-only and KG-only RAGs each use a single context. For a fixed generated answer, adding retrieved nodes can only increase the number of statements verifiable from c(q), giving the custom index a mechanical advantage on faithfulness that has nothing to do with retrieval quality. The paper does not control for this by, for example, retrieving the same total number of nodes from each index or evaluating faithfulness against a fixed reference context. This confound directly undermines the 'lowest hallucination rates' conclusion in Figure 3(c) and the accompanying text.","section":"RAG evaluation, Eq. (1)"},{"comment":"The evaluation reports only bar charts with error bars and no numeric means, standard deviations, confidence intervals, or statistical tests. With 2,295 queries, a claim of 'consistent' superiority requires at least a table of means and standard deviations and some pairwise significance testing (e.g., paired bootstrap or a mixed-effects model). Without these, the reader cannot determine whether the visible differences are meaningful or within sampling noise. This is load-bearing because the entire central claim of the paper is the superiority of the custom index.","section":"Results, Figures 3 and A1-A7"},{"comment":"All three metrics use GPT-4 as the 'gold' LLM judge, and no human calibration or agreement analysis is reported. The judge is from the same model family as the generator, and LLM judges are known to be biased by response length and fluency, which is especially concerning for the custom RAG whose answers are synthesized from a larger retrieved context and may therefore be longer. The qualitative feedback from CPG researchers mentioned later is not connected to the quantitative metrics, so it does not serve as validation. The paper should report a human-judged sample or at least an inter-annotator agreement study to establish that the metric scores track actual correctness, relevance, and faithfulness.","section":"Evaluation metrics, 'RAG evaluation'"},{"comment":"The conclusion states that the custom RAG offers 'superior performance at small compromise of speed,' but no runtime or latency measurements appear anywhere in the paper. Since the speed/quality trade-off is part of the summarized contribution, the absence of any timing data leaves this part of the claim unsubstantiated.","section":"Conclusion"}],"minor_comments":[{"comment":"There are several typos in this section: 'There metrics are defined' should be 'These metrics are defined', and 'an response' in the custom-index description should be 'a response'.","section":"RAG evaluation, 'Evaluation metrics'"},{"comment":"The terminology for the KG-based index is inconsistent: the text and figures use 'Knowledge Index', 'KG index', and 'Knowledge Graph index' interchangeably. Please standardize the terminology.","section":"Throughout"},{"comment":"The phrase 'norm number of keywords' appears to be a typo for 'normal number of keywords'.","section":"Appendix, Figures A4 and A5"},{"comment":"The citation of Touvron et al. (2023) for the LlamaIndex CorrectnessEvaluator is questionable; the evaluator is a LlamaIndex component, and the paper should cite the LlamaIndex documentation or a more directly relevant source for the evaluation modules.","section":"References"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a respectable, honest system report about an internal GenAI research assistant, but its central evaluation claim is weaker than the bar charts suggest. The stress-test note is right: the faithfulness metric is mechanically biased in favor of the hybrid index.\n\nWhat's genuinely useful here is the engineering integration. The paper describes a working MVP that connects three functions (research report generation, chat with documents, chat with multimodal data) around persistent knowledge bases that store both vector and knowledge-graph indices. That shared-KB design is a real usability win for a corporate research team, and the authors are transparent that they built it from open-source components (LangChain, LlamaIndex, Tavily, GPTResearcher) rather than claiming algorithmic novelty. The code is on GitHub, which is more than many such papers offer.\n\nThe evaluation, though, does not support the headline that the custom index 'generally offers superior performance.' The stress-test note identifies the core problem: in the custom RAG, the retrieved context is the union of the vector and KG retrievals, so the faithfulness score F = |V|/|S| has more context to verify statements against. More retrieved nodes can only increase |V| without increasing |S|, so the hybrid has a mechanical advantage that has nothing to do with retrieval quality. The same union effect can inflate correctness and relevance if the LLM judge rewards longer or more detailed answers. The paper reports only bar charts with error bars; no means, standard deviations, confidence intervals, or statistical tests appear anywhere. And the judge is GPT-4, uncalibrated against human ratings. The Smith et al. 2008 dataset and the manual difficulty/keyword labels are plausible, but they are not validated.\n\nNone of this suggests bad faith. The paper is careful to say it is not an algorithmic contribution, and its limitations section is candid about deployment challenges. The promotional language ('disruptive', 'unprecedented speed, magnitude and scope') is overblown for an internal MVP, and the qualitative claim about speeding up research is anecdotal. But the core problem is the confounded evaluation, which a serious referee would catch immediately.\n\nWho is this for? Practitioners building internal RAG tools, and researchers interested in evaluation pitfalls. It is not a scientific discovery. It deserves peer review only if the authors provide numeric results, statistical tests, the query set, and ideally a faithfulness evaluation that controls for context size (e.g., evaluating faithfulness on the same retrieved context for all methods). I'd accept the paper for review with that expectation, but I would not accept it as-is.","headline":"A transparent system report on an internal GenAI research assistant; the engineering is real, but the faithfulness metric gives the hybrid index an unfair mechanical advantage.","tokens_in":10554,"tokens_out":2463,"would_cite":false,"duration_ms":22435,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ScienceSage shows that a RAG querying a custom index that combines vector embeddings and a knowledge graph outperforms either index alone on correctness, relevance, and faithfulness across easy, medium, and hard queries.","keywords":["retrieval-augmented generation","knowledge graph index","vector index","hybrid retrieval","LLM-based evaluation","research report generation","knowledge base","multimodal RAG"],"falsifier":"Take a random sample of the 2,295 queries, have three human raters score the vector, knowledge-graph, and custom-index answers on the paper's three metrics, and compare the rank order; if the custom index is not first on a majority of metrics in the human-scored sample, the reported superiority is an artifact of the automated judge.","tokens_in":9526,"feed_emoji":"🤖","tokens_out":6660,"duration_ms":59586,"temperature":0.7,"pith_summary":"ScienceSage is a web application that lets researchers build a persistent knowledge base from their own documents, from images, audio, and video, and from reports generated from the latest internet information, then query that knowledge base in three ways: structured research report, document chat, and multimodal chat. The paper's central empirical claim is about the retrieval engine underneath those interfaces: a custom index that combines a vector embedding index with a knowledge graph index produces better answers than either index alone. On a test set of 2,295 questions over 85 documents, spanning easy, medium, and hard difficulty at three keyword-occurrence levels, the hybrid index scored highest on correctness, relevance, and faithfulness, with only a small speed compromise. The reason to care is practical: if the claim holds, researchers can get more accurate, better-grounded answers from their own corpora by combining two existing retrieval techniques rather than waiting for a new AI algorithm.","feed_headline":"Hybrid vector-plus-graph retrieval wins across 2,295 RAG queries","feed_subtitle":"Combining embeddings with a knowledge graph raises correctness, relevance, and faithfulness at small speed cost.","key_machinery":"The central object is the custom index: a hybrid retrieval structure in which one knowledge base is indexed twice, once as vector embeddings (numerical 'digital fingerprints' capturing semantic context) and once as a knowledge graph of subject-predicate-object triples, and the RAG queries both indexes, merges their retrieved contexts, and then synthesizes a response. The vector index does the work of broad semantic matching; the knowledge graph does the work of entity-anchored factual grounding; the merging step is what lets the two strengths combine. The evaluation machinery is a set of LLM-based evaluators from LlamaIndex, with GPT-4 as a 'gold' judge: correctness is scored against a reference answer on a 1-5 scale, and relevance and faithfulness are computed as ratios of relevant or context-supported statements in the response.","core_discovery":"The paper's central claim is that a retrieval-augmented generation (RAG) system that queries a custom index retrieves separately from a vector index and a knowledge graph index, concatenates the retrieved contexts, and only then synthesizes an answer, and that this hybrid scheme is consistently more correct, more relevant, and more faithful than either component alone. The authors attribute the effect to complementarity: the vector index supplies broad semantic context, while the knowledge graph anchors responses in explicit entities and relations, so the hybrid avoids the factual drift of pure semantic search and the narrowness of pure graph lookup. In their numerical experiment the pattern holds for easy, medium, and hard queries and after aggregation across keyword-occurrence levels, and the custom index also shows the lowest hallucination rates in the harder query regimes. They describe the cost as a small compromise in speed, which is why the conclusion is framed as a practical engineering choice rather than a new algorithm.","pith_inferences":["The reported hybrid advantage has not been separated from the LLM judge's possible preference for longer or more fluent answers; a human-rated subset or a downstream task like fact extraction would tell whether the gain is in retrieval or in presentation.","If the pattern generalizes beyond the Lincoln-document test set, hybrid retrieval could improve domain question answering on scientific and patent corpora without any change to the underlying language model; that is a cheap, model-agnostic extension worth testing.","The paper does not calibrate its GPT-4 judge against human annotations, so an immediate check is to rerun a random sample of the 2,295 queries with human raters and compare rank order."],"forward_implications":["RAG deployments that need both semantic breadth and factual grounding should prefer a hybrid vector-plus-graph index over a pure vector index, since the paper finds it higher on correctness, relevance, and faithfulness at a small speed cost.","Storing generated research reports into the same knowledge base that feeds document chat lets later queries draw on freshly retrieved internet information without a separate ingestion step.","For easy queries with clear keywords, the vector index alone is competitive; the knowledge graph index is the stronger fallback for entity-heavy medium and hard queries, and the custom index combines both advantages.","A single persistent knowledge base can serve three query interfaces—structured report generation, document chat, and multimodal chat—so a researcher can accumulate knowledge incrementally and reuse it across tasks."],"supporting_citations":[{"why":"LlamaIndex is the library used to build the three RAG variants and supplies the CorrectnessEvaluator, RelevancyEvaluator, and FaithfulnessEvaluator that produce the paper's scores.","marker":"Liu 2022"},{"why":"Supplies the 'combining vector spaces and knowledge graphs' rationale that the custom index is built on.","marker":"Mittal, Joshi, and Finin 2017"},{"why":"Provides the question-generation dataset whose documents and 2,295 queries (with difficulty and keyword-occurrence labels) form the testbed for the RAG comparison.","marker":"Smith, Heilman, and Hwa 2008"},{"why":"Defines the embedding representation used by the vector index.","marker":"Zamani and Croft 2016"},{"why":"Defines the knowledge graph triple structure and top-k search that the KG index relies on.","marker":"Yang et al. 2016"},{"why":"Introduces retrieval-augmented generation, the paradigm the RAG component implements.","marker":"Guu et al. 2020"}],"fun_headline_variants":["Vector+graph retrieval beats single-index RAG","Hybrid vector-graph RAG improves accuracy and faithfulness","Combining vector and graph retrieval reduces RAG hallucinations","ScienceSage's hybrid RAG outperforms pure vector or graph","Hybrid RAG offers accuracy gains at small speed cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4 'gold' judge gives trustworthy correctness, relevance, and faithfulness scores and that the manually labeled difficulty and keyword-occurrence categories on the 2008 question dataset track genuine retrieval difficulty; neither premise is calibrated against human judgments.","fun_headline_variants_meta":{"raw":{"variants":["Vector+graph retrieval beats single-index RAG","Hybrid vector-graph RAG improves accuracy and faithfulness","Combining vector and graph retrieval reduces RAG hallucinations","ScienceSage's hybrid RAG outperforms pure vector or graph","Hybrid RAG offers accuracy gains at small speed cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001511,"raw_usage":{"total_tokens":6037,"prompt_tokens":908,"completion_tokens":5129,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":5049}},"tokens_in":524,"tokens_out":5129,"duration_ms":35117,"temperature":1.0,"reasoning_tokens":5049,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:47:54.185461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 2,295 queries, have three human raters score the vector, knowledge-graph, and custom-index answers on the paper's three metrics, and compare the rank order; if the custom index is not first on a majority of metrics in the human-scored sample, the reported superiority is an artifact of the automated judge.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LlamaIndex is the library used to build the three RAG variants and supplies the CorrectnessEvaluator, RelevancyEvaluator, and FaithfulnessEvaluator that produce the paper's scores."},{"cited_title":"Thinking, Fast and Slow: Combining Vector Spaces and Knowledge Graphs","cited_arxiv_id":"1708.03310","evidence_quote":"Supplies the 'combining vector spaces and knowledge graphs' rationale that the custom index is built on."},{"cited_title":"A.; Heilman, M.; and Hwa, R","cited_arxiv_id":null,"evidence_quote":"Provides the question-generation dataset whose documents and 2,295 queries (with difficulty and keyword-occurrence labels) form the testbed for the RAG comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the embedding representation used by the vector index."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the knowledge graph triple structure and top-k search that the KG index relies on."}],"review_version":1}