{"id":"c061be51-5294-4140-a6dd-f7cb31840600","arxiv_id":"2608.13216","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Selectively grounding a design chatbot in a per-designer heterogeneous knowledge graph, with graph-transformer ranking of context, improved context retention and user-reported personalization in a nine-person study and on four LLM benchmarks.","lead":"CogChat builds a live knowledge graph from a designer's typed messages, then uses a Heterogeneous Graph Transformer to choose which remembered concepts the chatbot should draw on for each reply and each follow-up question. The authors report that grounding replies in this structured memory kept context across turns, cut conversational turns by roughly half, and lowered mental effort for nine professional designers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark KGs are built from reference/preferred responses (Sec. 4 Setup, Sec. 3.2.2); for ASQA/RewardBench this leaks answer entities into HGT-ranked context, so reported gains may reflect retrieval leakage rather than selective grounding.","rationale":"The central claim is that HGT-based entity selection outperforms ungrounded LLM interaction and naive KG augmentation. The technical evaluation is the primary direct evidence for this claim. The reader's weakest assumption identifies exactly the same vulnerability: the benchmark KGs are constructed from reference passages, and for ASQA and RewardBench those passages are the gold or preferred responses. This matters because the HGT retriever is trained on positive links derived from co-occurrence in those same passages and then selects context from the resulting graph. Under this setup, the top-k injected context can effectively contain the answer, while the Baseline never sees the passages. The reported gains then measure access to the answer key rather than the value of selective relational grounding. This concern is concrete and testable, and it affects the paper's main technical contribution. The user study, while internally consistent and honestly reported, cannot rescue the technical claim if the benchmark comparison is invalid. A CONDITIONAL verdict remains appropriate: the paper should be accepted only if recomputation without answer-derived KGs preserves the ordering. I do not move to REJECT because the leakage is not yet confirmed; the proposed test would settle it. The reader and I converge on the same load-bearing assumption, so my assessment agrees with the reader's weakest_assumption.","tokens_in":19427,"tokens_out":2755,"duration_ms":26127,"concrete_test":"Run RewardBench Chat Hard and Reasoning with KGs built from the query plus a neutral background corpus (e.g., Wikipedia passages on the topic) instead of the preferred response, with HGT trained only on query-side co-occurrence. If HGT's advantage over Baseline collapses or falls below about 5 points, the reported gains are leakage. As a second check, keep the current leakage-prone KG but replace HGT's top-k with k randomly sampled entities; if random top-k matches HGT, the result is gold-entity access, not selective ranking.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4 states: 'KGs were constructed per-query from questions and reference passages using the same extraction pipeline.' For ASQA and RewardBench, the only available 'reference passages' are gold long-form answers or preferred responses. Section 3.2.2 builds positive links from entity pairs co-occurring in 'the same utterance'; during evaluation this includes the gold/preferred response's entities, which then become positive training links for the HGT. The HGT is retrained on these link sets and used to rank nodes for prompt injection (Sec. 3.2.4), so the top-k context can contain the very entities needed to answer, while Baseline (Table 1) receives no passages. The largest deltas—Chat Hard +23.2 and Reasoning +22.4—occur exactly where preferred-response entities are most diagnostic. KG-only also contains the gold entities but injects them with noise; HGT's filtering to the gold entities explains the H>K ordering without any claim about relational grounding. This is a direct leak of the evaluation signal into both the retriever's training and its inputs, not merely a distribution shift. If it lands, the technical evaluation supports context access, not selective grounding, and the user study's attribution to HGT is undermined because the same pipeline supplies graph context in the study.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CogChat is a chat framework that constructs a personal heterogeneous knowledge graph from a designer's utterances, embeds it with a Heterogeneous Graph Transformer, and uses HGT-based ranking to select a small set of entities for prompt injection and to generate intentional and exploratory probing questions. The paper compares three conditions—Baseline (LLM-only), KG-only (naive full-graph injection), and KG+HGT (full pipeline)—on four technical benchmarks (ASQA, RewardBench, LoCoMo, LongMemEval) and in a within-subjects study with nine professional designers. The central claim is that selective relational grounding, rather than context capacity or naive graph augmentation, is what improves multi-turn design conversation and generalizes to ambiguous and long-horizon QA.","tokens_in":19537,"tokens_out":4557,"duration_ms":44276,"significance":"If the claims hold, the work is significant for UIST: it offers a concrete architecture that moves conversational context from recency-based token memory to a designer-specific structured relational memory, with a clean three-condition ablation and a mixed-methods user study. The interface features, especially in-context knowledge lookup and probing questions, are thoughtful and well-motivated by prior design-cognition literature. The paper also states its limitations candidly. However, the technical evaluation is seriously compromised by the way benchmark knowledge graphs are constructed from reference/gold passages, and the reported pairwise p-values in the user study are statistically impossible with nine participants. Both issues are load-bearing: the first affects the claim that HGT-based selection outperforms alternatives, and the second affects the reported significance of the user-study findings. These are correctable in principle, so the appropriate outcome is a major revision rather than rejection.","major_comments":[{"comment":"The benchmark knowledge graphs are built 'per-query from questions and reference passages using the same extraction pipeline' (Section 4, Setup). For ASQA and especially RewardBench, the only available reference passages are gold long-form or preferred responses. Section 3.2.2 creates positive links from entity pairs that co-occur 'within the same utterance,' so during evaluation the gold response's entities become positive training links for the HGT. Section 3.2.4 then ranks entities by cosine similarity between the query embedding and the HGT embeddings (Eq. 2) and injects the top-k into the prompt, while Baseline receives no passages. As a result, the large reported gains on Chat Hard (91.6 vs. 68.4) and Reasoning (94.2 vs. 71.8) may reflect answer leakage into both the retriever's training and its inputs, rather than evidence for selective relational grounding. I ask the authors to rerun the technical evaluation with knowledge graphs built only from the user-visible query and the model's own prior outputs, and to include an ablation that withholds entities appearing in the gold response from the graph, reporting how much of the HGT advantage remains.","section":"Section 4, Setup; Section 3.2.2; Section 3.2.4"},{"comment":"With n=9 participants, the minimum achievable two-sided exact p-value for a Wilcoxon signed-rank test is 2/512 = 0.0039. The paper repeatedly reports p<.001 for pairwise Wilcoxon tests (e.g., turn count, BERTScore, token-level IoU, negative feedback count, CUQ in Sections 6.1 and 6.2). These values cannot come from an exact signed-rank test; they appear to be artifacts of a normal approximation that is unreliable at this sample size. The authors should report exact permutation-based p-values, exact confidence intervals, or use small-sample-valid methods, and should verify that the reported p<.001 results survive the correction. This does not necessarily overturn the direction of the effects, but it invalidates the statistical significance claims as stated.","section":"Section 6, Analysis; Section 5.1"}],"minor_comments":[{"comment":"The text says HGT scores nodes by 'representation quality (embedding norm) and consistency with neighbors (inter-node similarity),' but Eq. (2) defines relevance only as cosine similarity between the query embedding and the node embedding. If norm or inter-node consistency affect ranking, they should be defined formally; if not, the sentence should be corrected.","section":"Section 3.2.4, Eq. (2)"},{"comment":"The continuity score includes a parameter alpha, but the manuscript never states its value or reports sensitivity analyses; please specify how alpha is chosen.","section":"Section 3.2.4, Eq. (3)"},{"comment":"The benchmark setup constructs KGs 'per-query,' whereas the deployed system updates and retrains the graph incrementally at every turn; the paper should clarify how the per-query benchmark KGs relate to the per-turn mechanism, including how many HGT training updates are run per query and whether the query's own reference passage is used during training.","section":"Section 4, Setup"},{"comment":"The phrase 'cognitive grounding' may overstate the contribution, since the paper explicitly disclaims modeling the designer's internal cognition; consider using 'relational grounding' or 'structured grounding' consistently to avoid confusion.","section":"Section 7, Discussion"},{"comment":"Several caption and text claims report p<.001 for pairwise Wilcoxon tests; after replacing them with exact small-sample tests, please update all significance symbols and any related effect-size statements consistently.","section":"Section 6 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark leakage and the impossible pairwise p-values are serious, but both are fixable within the scope of a revision: the technical evaluation can be re-run without gold-derived context, and the user-study statistics can be recomputed with exact small-sample methods. The system contribution and user-study design are valuable, and the limitations section is unusually candid, so I would not reject the paper outright. Please require the authors to address the leakage and the statistical reporting before considering it further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The integrated system is real and described in unusual detail—pipeline, prompts, ablation conditions, and a user study. The technical evaluation, however, is compromised: KGs are built from reference passages, which for ASQA and RewardBench are the gold or preferred responses, and the HGT is trained on links mined from those same passages and then used to select top-k context. So the benchmark gains, especially Chat Hard (+23.2) and Reasoning (+22.4), likely reflect retrieval leakage rather than selective grounding.\n\nWhat is genuinely new is the integration: a live per-designer heterogeneous KG, HGT-based context selection, and embedding-zone probing for design conversation. Each component is prior art, but the combination is new. The user study is small (n=9) but internally consistent—effect directions align across self-report, behavioral, and interview data. The authors also state their limitations openly, including the shared probing components and the need for targeted ablations. That honesty is real.\n\nThe main soft spot is the benchmark setup. Section 4 says KGs were constructed per-query from questions and reference passages. For RewardBench and ASQA, the only reference passages are the preferred/gold responses. Positive links come from entity co-occurrence in the same utterance (Section 3.2.2), so during evaluation the gold response's entities become positive training links for the HGT, which then ranks those entities into the top-k context. Baseline receives no passages at all. The H>K>B ordering is exactly what you would predict from answer leakage. This does not sink the user study, which uses real designer conversations with no gold answers, but it does mean the technical eval cannot support the paper's central claim that HGT's relational selection—not simply fetching the answer's entities—drives the gains. The user study also cannot fully isolate the mechanism, as the authors concede. Smaller issues: no code or data shipped, some p<.001 claims with n=9 and no exact test details, and the KG-only condition in the user study may not match KG+HGT in context volume.\n\nWho should read this: researchers working on knowledge-graph-grounded conversational tools and on evaluation leakage in RAG systems. It deserves a serious referee—it is a competent systems contribution with a fixable flaw. The fixes are straightforward: specify baseline passage access, build KGs only from the question and prior turns (no gold responses), compare against a memory-agent baseline, and release artifacts. If the leakage is addressed, the results become meaningful. My recommendation: send it to peer review with the expectation of major revision.","headline":"A well-built system with an honest limitations section, but the benchmark evaluation leaks the answer into the HGT's training and selection, so the headline claim about selective grounding isn't supported by the technical eval.","tokens_in":20258,"tokens_out":3950,"would_cite":false,"duration_ms":33443,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that grounding conversational AI in a per-designer heterogeneous knowledge graph, with a Heterogeneous Graph Transformer selecting which nodes to inject, outperforms both ungrounded LLM interaction and naive full-graph…","keywords":["knowledge graph","heterogeneous graph transformer","conversational AI","design cognition","context grounding","probing questions","personalization","long-term dialogue memory"],"falsifier":"Re-run the four benchmarks with graphs built only from the question and prior conversation turns, never from the reference or preferred answer. If the KG+HGT advantage over both Baseline and KG-only collapses or drops substantially, the reported gains came from exposing the preferred response's entities rather than from selective relational grounding. For the user study, a blind expert rating of the final design artifacts across conditions would settle whether the self-reported depth translates into output quality.","tokens_in":19054,"feed_emoji":"💬","tokens_out":5806,"duration_ms":53147,"temperature":0.7,"pith_summary":"CogChat builds a personal heterogeneous knowledge graph from each designer's utterances, extracts typed entities and relations, and uses a Heterogeneous Graph Transformer (HGT) to choose which graph nodes are injected into the LLM's prompt at each turn. The paper argues that preserving relational context requires structure, not capacity: exhaustive graph injection (KG-only) overdraws on noise, while HGT-selected top-k grounding (KG+HGT) beats both unfiltered graph memory and the ungrounded baseline. Technical evaluation on ASQA, RewardBench, LoCoMo, and LongMemEval reports KG+HGT highest on all four, with the largest gaps on ambiguous and relationally complex subsets. A within-subjects study with nine professional designers reports fewer turns, more consistent vocabulary reuse, higher probing success, lower reported cognitive load, and deeper engagement for the HGT-grounded condition. If correct, the central consequence is that the bottleneck in multi-turn design conversation is context curation, deciding which relations to foreground, rather than context-window capacity.","feed_headline":"Selective graph grounding beats raw LLM memory in design chat","feed_subtitle":"CogChat shows that retaining relational structure, not context capacity, is what keeps design conversation coherent.","key_machinery":"The load-bearing object is the Heterogeneous Graph Transformer (HGT), a graph neural layer with type-specific query, key, and value projections and relation-specific attention, used to embed a heterogeneous knowledge graph whose nodes carry types such as CONCEPT, DESIGN_ELEMENT, MATERIAL, PROPERTY, and ACTION, initialized from CLIP embeddings. The graph is built per turn via a GraphRAG-style extraction pipeline; positive and negative link sets are sampled from co-occurrence and designer feedback, and the HGT is retrained incrementally on them. At response time, nodes are ranked by cosine similarity between the current message embedding and HGT node embeddings, and the top roughly twenty entities, with their relations, enter the LLM prompt. That selection step is what the paper contrasts against KG-only, which injects all extracted entities, to isolate selective grounding as the cause of the observed gains.","core_discovery":"The central claim is that grounding a conversational LLM in a selectively injected, relationally structured personal knowledge graph improves multi-turn design conversation over both no grounding and unfiltered graph grounding. The paper's own framing is that recency-based context decays cross-turn relations no matter how large the window, and that injecting the full graph adds noise that competes with salient nodes; HGT-based ranking fixes that by foregrounding structurally relevant entities. Because the KG+HGT condition outperforms KG-only on every tested benchmark, and because the KG-only versus KG+HGT gap is significant in the user study on semantic similarity, token overlap, probing success, and task time, the paper concludes that selective graph-grounded context injection, not memory capacity, is the operative mechanism. The paper further claims that the same ordering generalizes from ambiguity and preference benchmarks to long-term memory benchmarks, motivating its use in design practice.","pith_inferences":["An implication the authors leave implicit is that any selective retriever, not specifically HGT, might reproduce part of the effect; a fair test would replace HGT with a simpler context selector, such as plain CLIP similarity without graph training, and compare the three-condition ordering.","Because the technical benchmarks construct per-query knowledge graphs from questions plus reference or preferred-answer passages, part of the reported edge may reflect entity overlap with the gold response; my reading is that the benchmark numbers are strong evidence for the value of relevant context access, but weaker evidence for relational grounding per se.","A testable extension is to run the same three-condition comparison on a multi-session design project spanning days or weeks, since the paper's own limitations note that graph decay, contradictory constraints, and intent pivots remain open challenges that short single-session tasks cannot expose.","The probing feedback loop suggests a possible positive spiral: better questions produce denser graphs, and denser graphs produce better questions; if that holds, gains should compound across turns, a prediction the current 30-minute sessions only hint at."],"forward_implications":["Context-window growth alone will not fix multi-turn failure: the Baseline received full conversation history, yet HGT-selected context still won.","Naive knowledge-graph augmentation can hurt: KG-only injected noise that compounded errors on ambiguous queries, so adding graph memory without selection is not an improvement.","Selective grounding gives its largest gains on relationally dense or ambiguous tasks, with RewardBench improvements of 23.2 points on Chat Hard and 22.4 points on Reasoning, so the method should be expected to help wherever utterances are polysemous and relational.","Long-horizon dialogue benefits: the ranking held on LongMemEval and LoCoMo, indicating the mechanism transfers beyond design-style ambiguity to multi-session recall.","In user interaction, the graph-grounded system cut turns to completion by 45.9%, nearly doubled token-level vocabulary reuse relative to KG-only, raised probing question answer rates from 0.17 to 0.70, and lowered reported mental demand, effort, and frustration."],"supporting_citations":[{"why":"Defines the Heterogeneous Graph Transformer used to embed the graph and rank entities, the mechanism whose selective grounding the paper evaluates.","marker":"[15]"},{"why":"Supplies the GraphRAG entity-relation extraction pipeline that builds each per-turn heterogeneous knowledge graph.","marker":"[9]"},{"why":"ASQA, the ambiguous long-form question answering benchmark used as one of the four evaluation tasks.","marker":"[35]"},{"why":"RewardBench, the preference alignment benchmark where the largest gains from selective grounding appear.","marker":"[19]"},{"why":"LoCoMo, the very long-term conversational memory benchmark used to test transfer beyond design-style tasks.","marker":"[24]"},{"why":"LongMemEval, the long-term interactive memory benchmark used to test transfer beyond design-style tasks.","marker":"[41]"},{"why":"Documents the multi-turn performance degradation of LLMs that the paper identifies as the problem CogChat targets.","marker":"[18]"}],"fun_headline_variants":["Selective graph beats raw LLM memory in design chat","HGT-filtered graph grounding beats raw LLM memory","Relational context, not capacity, keeps design chat coherent","CogChat: Structure-aware AI keeps design chats on track","Graph-grounded chat: HGT filtering beats full memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The technical results stand or fall on whether per-query knowledge graph construction from questions plus reference passages produces the same signal a live designer's graph would, with no answer leakage; the user-experience results additionally rest on a nine-person, single-session study that the authors themselves caution is not evidence of long-term personalization.","fun_headline_variants_meta":{"raw":{"variants":["Selective graph beats raw LLM memory in design chat","HGT-filtered graph grounding beats raw LLM memory","Relational context, not capacity, keeps design chat coherent","CogChat: Structure-aware AI keeps design chats on track","Graph-grounded chat: HGT filtering beats full memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3073,"prompt_tokens":966,"completion_tokens":2107,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":2025}},"tokens_in":582,"tokens_out":2107,"duration_ms":25421,"temperature":1.0,"reasoning_tokens":2025,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:13:57.922836+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four benchmarks with graphs built only from the question and prior conversation turns, never from the reference or preferred answer. If the KG+HGT advantage over both Baseline and KG-only collapses or drops substantially, the reported gains came from exposing the preferred response's entities rather than from selective relational grounding. For the user study, a blind expert rating of the final design artifacts across conditions would settle whether the self-reported depth translates into output quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ASQA, the ambiguous long-form question answering benchmark used as one of the four evaluation tasks."}],"review_version":1}