{"id":"a5ff212d-38fb-4003-b35a-12a0a31e768e","arxiv_id":"2605.27176","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Empirical perturbations of local KGs across three LLMs reveal that compact top-k and other subsets often recover most of the hypothesis-generation signal, supporting a compressive KG hypothesis.","lead":"Experiments on three LLMs show compact subgraphs from knowledge graphs can produce similar hypothesis-generation behavior as full graphs for battery materials, even when key triples are held out. Smart generalists might read it to understand how much structured scientific context AI systems actually need versus what they already encode in their training priors.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"No-KG baselines recovering substantial graph content indicates metrics may attribute prior knowledge to provided subgraphs.","rationale":"The reader's weakest assumption directly identifies the confounding risk from priors and metric artifacts; the abstract's own admission that no-KG recovers graph content makes this the single most load-bearing point for the central claim. No other internal inconsistency is visible from the provided material, and the paper does not claim formal verification or parameter-free derivation that would override the empirical concern.","tokens_in":1693,"tokens_out":340,"duration_ms":23942,"concrete_test":"Quantify the fraction of fixed-reference and provided-graph score that is already achieved in the no-KG condition; subtract this baseline from all other conditions and re-test whether compact subgraphs still match full-KG performance within 5% after subtraction. If the gap closes, the compressive claim is not supported beyond priors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The compressive hypothesis requires that differences in hypothesis quality (via provided-graph and fixed-reference metrics) can be causally attributed to the presence/absence of specific perturbed graph facts rather than model priors. The abstract explicitly states that no-KG outputs recover substantial graph content from priors, yet the evaluation still treats metric improvements under compact subgraphs as evidence of recoverable KG signal. If the fixed-reference metric scores hypotheses against known scientific outcomes (which LLMs may have memorized) or if provided-graph rewards surface overlap with any supplied triples, then top-k, random, and topology-based subsets appearing equivalent to full KGs could simply reflect that all conditions are dominated by the same prior knowledge rather than demonstrating redundancy within the KG itself.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper examines which facts from local knowledge graphs (KGs) matter for LLM-based scientific hypothesis generation in battery materials. Using Mistral-7B, Llama-3.1-70B, and Gemini 2.5 Flash, it perturbs KGs along axes of density, ontology richness, topology, and control structure, then evaluates hypothesis quality via provided-graph and fixed-reference metrics. Key findings are that KG utility is selective and model-dependent, no-KG baselines recover substantial graph content from priors, and compact top-k (or even random/topology-based) subgraphs often match full-KG performance even when claimed-outcome triples are held out. These observations are framed as support for a redundancy-aware \"Compressive Knowledge Graph Hypothesis.\"","tokens_in":1833,"tokens_out":485,"duration_ms":28139,"significance":"If the perturbation results and metric comparisons prove robust after proper controls, the work would indicate that scientifically structured KGs contain substantial redundancy for hypothesis generation tasks, allowing compact subgraphs to substitute for full local graphs. This could inform more efficient KG-augmented LLM pipelines in materials science and related domains. The empirical framing (rather than axiomatic) and explicit acknowledgment of prior leakage are strengths, though the absence of reported quantitative outcomes limits immediate impact assessment.","major_comments":[{"comment":"Abstract: The experimental design and headline outcomes are described, but the abstract supplies no quantitative results, statistical tests, exact definitions of the provided-graph and fixed-reference metrics, sample sizes, or controls for model-prior leakage. The central compressive hypothesis therefore rests on unshown experimental details whose soundness cannot be evaluated from the given text.","section":"Abstract"},{"comment":"Abstract (perturbation and evaluation description): The claim that differences in hypothesis quality can be causally attributed to specific perturbed graph facts (rather than model priors) is load-bearing for the compressive hypothesis, yet the text notes that no-KG outputs already recover substantial graph content. Without reported controls (e.g., ablation of metric overlap with memorized scientific outcomes or surface-triple matching), it is unclear whether top-k/random/topology subsets appearing equivalent to full KGs demonstrates KG-internal redundancy or simply prior dominance across all conditions.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive report. We address each major comment below and indicate where revisions will be made to the manuscript.","responses":[{"response":"We agree that the abstract is currently high-level and omits key quantitative and methodological details. In the revised version we will expand the abstract to include headline quantitative outcomes (e.g., performance of top-k and other subsets relative to full KG and no-KG baselines), brief definitions of the provided-graph and fixed-reference metrics, sample sizes, and explicit reference to the no-KG condition as a control for prior leakage. This will make the central claims evaluable directly from the abstract.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The experimental design and headline outcomes are described, but the abstract supplies no quantitative results, statistical tests, exact definitions of the provided-graph and fixed-reference metrics, sample sizes, or controls for model-prior leakage. The central compressive hypothesis therefore rests on unshown experimental details whose soundness cannot be evaluated from the given text."},{"response":"This concern about disentangling graph contributions from priors is well-taken. The manuscript already reports a no-KG baseline that quantifies prior recovery and employs a fixed-reference metric that scores against established scientific outcomes. The key comparative result is that compact subsets match full-KG performance while both exceed the no-KG baseline, even when claimed-outcome triples are held out; this pattern is what supports the compressive hypothesis. We did not, however, include explicit ablations for metric overlap with memorized content beyond the no-KG condition. We will strengthen the discussion section to address this potential confound more explicitly and will add a targeted control experiment if feasible within the revision timeline.","revision_made":"partial","referee_comment":"[Abstract] Abstract (perturbation and evaluation description): The claim that differences in hypothesis quality can be causally attributed to specific perturbed graph facts (rather than model priors) is load-bearing for the compressive hypothesis, yet the text notes that no-KG outputs already recover substantial graph content. Without reported controls (e.g., ablation of metric overlap with memorized scientific outcomes or surface-triple matching), it is unclear whether top-k/random/topology subsets appearing equivalent to full KGs demonstrates KG-internal redundancy or simply prior dominance across all conditions."}],"tokens_in":1437,"tokens_out":499,"duration_ms":60453,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core observation is that top-k, random, and topology-based subsets of the local KG produce hypothesis outputs that track full-KG behavior across the three models tested, and this holds even when claimed-outcome triples are held out. The paper runs a clean set of perturbations on density, ontology richness, topology, and controls, then scores with both a provided-graph metric and a fixed-reference metric. That gives a practical data point for anyone trying to shrink KGs in LLM pipelines without losing much on this narrow battery-materials task.\n\nThe experiments are straightforward and the model-dependence finding is useful. They also flag upfront that no-KG runs already recover substantial graph content from priors, which is honest.\n\nThe soft spot is exactly the one the stress-test raises. If the fixed-reference metric is comparing against known scientific outcomes the models have likely seen in training, and the provided-graph metric rewards surface overlap with whatever triples are supplied, then the apparent equivalence between random subsets and semantic ones could just mean all conditions are dominated by the same prior knowledge. The abstract states the prior recovery explicitly, yet the compressive hypothesis is still framed as KG signal being recoverable from compact subgraphs. Without seeing the actual numbers, statistical tests, or any ablation that holds the model prior fixed, it's difficult to know how much the metrics are isolating the contribution of the supplied facts.\n\nThis is the sort of incremental empirical note that matters to groups already running KG-augmented generation in materials or chemistry. It is worth a serious referee because the question is well-posed and the setup is replicable, even if the interpretation around priors needs tightening.","headline":"Compact subgraphs recover similar hypothesis signal to full KGs here, but the metrics look vulnerable to model priors doing the work.","tokens_in":2339,"tokens_out":399,"would_cite":false,"duration_ms":26082,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Compact scientifically structured subgraphs recover most useful signal from full knowledge graphs for hypothesis generation.","keywords":["knowledge graphs","hypothesis generation","battery materials","compressive hypothesis","subgraph selection","model priors","language models","graph perturbations"],"falsifier":"If compact top-k or random subgraphs consistently produce lower-quality hypotheses than full local graphs on the same models and materials when measured by the provided-graph metric, the compressive hypothesis would be falsified.","tokens_in":2599,"feed_emoji":"🔬","tokens_out":677,"duration_ms":33140,"temperature":0.7,"pith_summary":"The paper tests which facts in local knowledge graphs actually drive language-model outputs when generating hypotheses about battery materials. Across three models it perturbs graphs along density, ontology richness, topology, and control structure, then measures quality with both provided-graph and fixed-reference metrics. Results show that model outputs shift with added graph context yet still recover substantial content from internal priors even without any graph. Compact top-k subgraphs frequently match full-graph performance, and the same holds for random or topology-based subsets. This pattern leads the authors to the compressive KG hypothesis that useful scientific signal is often redundant and recoverable from small structured subsets rather than the entire local graph.","feed_headline":"Compact subgraphs recover most KG signal for hypothesis generation","feed_subtitle":"Tests across three models show top-k and random subsets often match full-graph performance, pointing to redundancy in graph facts.","key_machinery":"Perturbations of local KGs along density, ontology richness, topology, and control structure, scored by provided-graph and fixed-reference metrics, that isolate the contribution of specific graph facts to hypothesis quality.","core_discovery":"Across Mistral-7B, Llama-3.1-70B, and Gemini 2.5 Flash, KG-guided hypothesis generation for battery materials exhibits selective and model-dependent utility; graph context alters outputs, yet no-KG baselines already recover much of the same content from model priors, while compact top-k subgraphs (including when claimed-outcome triples are held out) approximate full-KG behavior and this recovery is not restricted to semantic ranking rules since random and topology-based subsets also recover substantial signal.","pith_inferences":["If the pattern holds, KG construction for hypothesis generation could prioritize high-signal subgraphs over exhaustive local coverage.","The same redundancy may appear in scientific domains beyond battery materials where models already encode substantial domain knowledge internally.","A direct test would apply the same density and topology perturbations to KGs drawn from other scientific fields and compare recovery rates."],"forward_implications":["KG utility remains selective and model-dependent even after perturbations.","No-KG outputs already recover substantial graph content from model priors.","Compact top-k subgraphs approximate full-KG behavior including with held-out triples.","Signal compression is not unique to semantic ranking; random and topology-based subsets recover much of the same signal."],"fun_headline_variants":["Compact subgraphs suffice for KG-guided hypotheses","Redundant facts mean no need for full local KGs","Model priors recover graph content in no-KG cases","Topology and random subsets both approximate full graphs","Compressive KG hypothesis tested on battery materials"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The perturbation operations and the two evaluation metrics isolate the causal contribution of specific graph facts to hypothesis quality without substantial confounding from model priors or metric artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Compact subgraphs suffice for KG-guided hypotheses","Redundant facts mean no need for full local KGs","Model priors recover graph content in no-KG cases","Topology and random subsets both approximate full graphs","Compressive KG hypothesis tested on battery materials"]},"model":"grok-4.3","cost_usd":0.004837,"raw_usage":{"total_tokens":2290,"prompt_tokens":657,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":48365500,"prompt_tokens_details":{"text_tokens":657,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1564,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":657,"tokens_out":69,"duration_ms":18577,"temperature":1.0,"reasoning_tokens":1564,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T16:56:23.944525+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If compact top-k or random subgraphs consistently produce lower-quality hypotheses than full local graphs on the same models and materials when measured by the provided-graph metric, the compressive hypothesis would be falsified.","supporting_citations":[],"review_version":1}