{"id":"a8a06230-55c3-4b02-9149-2bdc9ecb1458","arxiv_id":"2508.13675","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"On household task knowledge graphs, standard embedding-based link prediction models are outperformed by frequency baselines and an LLM, suggesting these graphs break common LP assumptions.","lead":"This paper builds knowledge graphs of household chores from video annotations and tests whether standard link prediction algorithms can guess the overall task or the next step. They find that simple frequency rules and a large language model beat the graph embedding methods on this benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set entities are absent from the transductive embedding models' entity tables; near-zero KGE scores may be an artifact of the split rather than evidence about situational graph structure.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the paper evaluates transductive embedding models on test components whose entity nodes are absent from the training graph, and never explains how those unseen nodes receive embeddings. This is not a minor methodological gap; it is the foundation of the paper's headline result. If the test entities are represented only by untrained random embeddings, then the near-zero Hits@k values are expected regardless of graph structure, and the comparison to frequency baselines and LLMs is invalid. The paper provides no inductive baseline, no dense-graph control, and no details of the ranking protocol, so the reader's REJECT verdict is appropriate. I would not adjust the verdict; the concern is serious and currently unresolved. A code-level check can settle it, but until then the central claim remains unsubstantiated.","tokens_in":7209,"tokens_out":7902,"duration_ms":88339,"concrete_test":"Inspect the released code (github.com/marusta/Knowledge_Refinement) to identify how the PyKeen TriplesFactory is built for test components. Specifically, print the entity IDs in the training factory and the test factory and count the overlap. Then run one held-out test triple (e.g., a 'has next' edge from a test component) through the trained TransE model. If the test entities are absent from the trained entity embedding matrix, the published Hits@k values cannot be reproduced under standard transductive evaluation; if they are present, check whether their embeddings were trained or randomly initialized. A cleaner analytic check: re-score the test components using only training entities mapped by action/object labels; if Hits@1 for sub-action changes materially from 0%/52%, the reported results are protocol artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that situational KGs defeat standard LP models. That claim rests on Table 3, which reports near-zero Hits@k for TransE/ComplEx/RotatE on held-out test components. But the paper's own split (§3) puts 108 entire components (2,840 nodes) into test, disjoint from the 11,088 training nodes. TransE, TransR, ComplEx, DistMult, and RotatE are transductive: they learn an embedding per entity ID and have no mechanism to score unseen nodes. The paper never states how test entities obtain embeddings. If they were added with untrained/random initializations—or if the reported ranking was computed over a small candidate set instead of the full entity set—then the near-zero scores are a mechanical consequence of the protocol, not of 'disconnected subgraphs' or 'hierarchical dependencies.' This is load-bearing because the entire negative result and the LLM/baseline superiority conclusion depend on those numbers. The paper's own future-work call to 'redefine evaluation protocols for LP in disconnected graphs' does not repair the current comparison; an invalid baseline comparison cannot establish the headline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies knowledge graph completion for action prediction on a situational KG built from the KIT Bimanual Actions Dataset. It constructs one graph component per video recording, with parent actions, sub-actions, objects, and relations such as has_element and has_next. The authors compare embedding-based KGC models (TransE, TransR, ComplEx, DistMult, RotatE, and LiteralE variants), simple frequency baselines, and GPT-4o-mini on two tasks: predicting the parent action of an observed sub-action sequence and predicting the next sub-action. The reported results show near-zero Hits@k for KGE models on parent-action prediction, high performance of the object-aware frequency baseline, and strong GPT-4o-mini performance on the parent-action task. The paper concludes that standard LP methods are poorly suited to situational KGs due to disconnected subgraphs and hierarchical dependencies, and calls for new evaluation protocols.","tokens_in":7480,"tokens_out":4443,"duration_ms":52840,"significance":"If the empirical comparison were valid, the paper would provide a useful cautionary result for the robotics/KGC community: standard transductive embedding models may fail on fragmented, component-wise graphs, while simple heuristics and LLMs can be competitive. The paper also contributes a public dataset, a fixed train/test split, and reproducible code, which are valuable assets. However, the central negative result is undermined by a fundamental evaluation-protocol problem: the test components contain entities unseen during training, yet all compared KGE models are transductive and require an embedding for every scored entity. The paper never explains how test entities are represented. Because the headline conclusion rests entirely on Table 3, this issue is load-bearing, not a presentation detail.","major_comments":[{"comment":"The evaluation protocol is invalid as described for the KGE models. The paper states that two of ten repetitions per task were reserved for testing, yielding 11,088 training nodes and 2,840 test nodes, and that the split is held constant across models. TransE, TransR, ComplEx, DistMult, and RotatE learn an embedding per entity ID and have no mechanism to score entities that were absent from training. The paper never states how test entities receive embeddings. If they were randomly initialized or simply omitted, the near-zero Hits@k values in Table 3 are a mechanical artifact of the split rather than evidence about situational graph structure. The central claim — that standard LP models are 'not fit for the job' — is therefore unsupported by the reported numbers.","section":"Section 3 (train/test split) and Table 3"},{"comment":"The manuscript omits essential evaluation details: whether filtered or unfiltered Hits@k was used, how the candidate set was constructed, how many random seeds or training runs were averaged, and whether any hyperparameter tuning was performed. Without error bars or repeated trials, the reported differences among KGE models (e.g., RotatE 3.88% vs TransE 3.25% vs TransR 3.25% at Hits@1 in parent-action prediction) are not interpretable. The sub-action results also contain an anomaly: TransE and TransR report Hits@1 = 0.00% but Hits@3 = 53.40% and 31.51%, respectively, which is difficult to reconcile with a standard filtered ranking protocol and suggests either very small candidate sets or an implementation inconsistency. The paper must specify the exact protocol before any comparison can be trusted.","section":"Section 3 (PyKeen setup) and Table 3"},{"comment":"The LLM comparison is not on equal footing with the KGE models. The prompt in Figure 2 asks GPT-4o-mini to answer with exactly one of nine predefined parent-action labels, making it a nine-way classifier. The frequency baselines also map to the same nine labels. In contrast, the KGE link-prediction models are scored by ranking against the full entity set, which includes thousands of possible tails. This mismatch alone can produce the reported performance gap: the LLM and baselines are evaluated on a closed-set classification task while the KGE models are evaluated on open-set link ranking. The paper should either restrict the KGE candidate set to the nine parent actions (and the corresponding sub-action vocabulary) or evaluate all methods under a common ranking protocol.","section":"Section 4 and Figure 2 (GPT-4o-mini comparison)"},{"comment":"The paper attributes the poor KGE performance to 'disconnected subgraphs' and 'hierarchical dependencies,' but the current experimental design cannot separate these structural explanations from the trivial transductive-entity-novelty problem. The conclusion is therefore overclaimed. A minimal control would be to evaluate inductive link-prediction models (e.g., GraIL, NBFNet) or to train the embedding models on the union graph with edge masking rather than on disjoint components. The future-work sentence in Section 5 that calls for 'redefined evaluation protocols' does not repair the current comparison, since the paper's headline result is precisely the one produced by the flawed protocol.","section":"Section 5 (conclusion)"}],"minor_comments":[{"comment":"Reference [13] and [14] are duplicates (the same survey by Ji et al.). This should be corrected. Reference [28] for GPT-4o-mini lacks a version/date and is incomplete.","section":"References"},{"comment":"Several cells use dashes where values could be expected (e.g., Random Hits@5, GPT-4o-mini Hits@3/Hits@5). Please indicate whether these are not applicable or not measured. Also, the table would benefit from confidence intervals or standard deviations.","section":"Table 3"},{"comment":"The statement that the data was 'structured to ensure compatibility with MemNet' is not explained, and MemNet is not otherwise used in the experiments. Either clarify or remove the mention.","section":"Section 3 (graph construction)"},{"comment":"The phrase 'special characteristics' is used as the paper's central explanation, but the paper never defines a formal set of distinguishing characteristics beyond disconnected components and hierarchy. A precise characterization would strengthen the argument.","section":"Abstract and Introduction"}],"recommendation":"reject","confidential_remarks":"The dataset and code release are commendable, and the paper addresses a relevant applied problem. However, the main empirical claim is built on an invalid evaluation protocol: transductive KGE models are scored on test components whose entities were never seen during training, and no method for handling unseen entities is described. The LLM/baseline comparison also uses a closed-set classification setup that is not comparable to open-set link ranking. These are not fixable by local edits; the experimental design would need to be substantially redone with inductive baselines and a common evaluation protocol. I recommend rejection, though I would encourage the authors to resubmit a revised comparison using a valid protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper is worth a look for the dataset and the benchmark framing, but the headline claim—that standard KGC models are unfit for situational graphs—rests on an evaluation that, as written, cannot be right. The test set is 108 whole graph components whose entity nodes never appear in training. TransE, ComplEx, RotatE, etc. are transductive: they only have embeddings for training entities. The paper never says how test entities get embeddings, so all the near-zero Hits@k numbers in Table 3 are plausibly a mechanical artifact rather than evidence about graph structure.\n\nWhat is new: they construct a temporal, hierarchical KG from the KIT Bimanual Actions dataset, with 'has next' and 'has element' relations, and they benchmark standard KGC models against simple frequency baselines and an LLM. That's a reasonable idea and the problem is relevant for robotics. The baselines themselves are sensible. The paper also releases code and data, which is good practice.\n\nThe soft spots: (1) the evaluation protocol is underspecified. If they trained on the train components and then evaluated on test components with no entity overlap, either the models should fail to run or they were scored with some ad-hoc handling of unseen entities. That's the load-bearing issue. (2) No error bars or repeated trials, which makes it hard to judge whether the small differences between models mean anything. (3) The LLM prompt gives the nine possible parent actions, so the comparison to KGE models is not apples-to-apples for open-set link prediction. That's a lesser concern, but worth noting. (4) Minor: reference [13] and [14] are duplicated.\n\nThe conclusion's call for 'redefining evaluation protocols for LP in disconnected graphs' reads like an admission that the current comparison is broken. A better paper would use an inductive link prediction model or would split the graph at the node level rather than the component level, so test entities appear in training but in different contexts.\n\nWho is this for? Researchers working on KGC for robotics; the paper is useful as a cautionary example and as a dataset release. But as a scientific claim, the current version doesn't support the conclusion. I'd send it to review only if the authors can fix the evaluation—right now, I'd recommend a major revision or reject-and-resubmit, with the transductive issue front and center.\n\nI'd probably not cite the results, but I'd keep the dataset in mind.","headline":"Useful dataset and benchmark framing, but the central negative result on KGE models is likely an artifact of a transductive evaluation on unseen test entities.","tokens_in":7907,"tokens_out":3730,"would_cite":false,"duration_ms":37756,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that situational knowledge graphs built from household action videos have a fragmented, temporal, hierarchical structure that breaks standard embedding-based link prediction, leaving simple frequency baselines and a few-sh","keywords":["knowledge graph completion","link prediction","situational knowledge graphs","household action prediction","frequency baselines","large language models","transductive embeddings"],"falsifier":"Retrain the embedding models with one held-out test component included in the training graph, or initialize unseen test entities with feature-based embeddings, and recompute Hits@k. If scores jump to the baseline range, the reported failure is an artifact of unseen-node evaluation rather than a property of situational graphs; if they stay near zero, the structural mismatch is confirmed.","tokens_in":7120,"feed_emoji":"🤖","tokens_out":9667,"duration_ms":83449,"temperature":0.7,"pith_summary":"The paper tests whether knowledge-graph completion can predict what a person is doing (the parent action) and what they will do next (a sub-action) from graphs built out of household-task video recordings. It finds that standard embedding-based link prediction models essentially fail on these situations: on parent-action prediction they stay below 6% Hits@5, while a frequency-counting baseline reaches 76-100% Hits@3 and a large language model reaches 78.73% Hits@1. On sub-action prediction, a simple transition-count baseline beats every model, including the language model. The paper's central explanation is that situational graphs are fragmented into one disconnected component per recording and carry hierarchical and temporal relations, which violates the dense-connectivity assumptions behind conventional link prediction benchmarks. This matters because household robots that assist people depend on exactly this kind of incomplete, situational knowledge.","feed_headline":"Simple rules beat KG embeddings for household action prediction","feed_subtitle":"On household-task videos, standard link prediction scores near zero while frequency counts and an LLM dominate","key_machinery":"The central object is the situational knowledge graph itself, built from a household-action video dataset with four relation types: has actor, has object, has element, and has next. Each of the 540 recordings becomes its own disconnected weak component, giving 13,928 nodes, 32,577 edges, very low density (0.00017), and zero reciprocity. A fixed train/test split, holding out two of ten repetitions of each task, is applied identically to every model. This graph carries the argument because it is the testbed that reveals the failure: conventional embedding models are transductive and cannot represent test components whose nodes never appeared in training, while frequency baselines and the LLM o","core_discovery":"The central claim is that situational knowledge graphs, which describe a household activity as a parent action decomposed into object-related sub-actions connected by has-next edges, have structural properties that defeat standard knowledge-graph completion. The graph built from 540 recordings of nine household tasks contains 13,928 nodes and 32,577 edges but forms 540 disconnected weak components, one per recording, with zero reciprocity. On parent-action prediction, the embedding models reach at most 5.72% Hits@5, whereas a context-aware frequency baseline reaches 76% Hits@1 and 100% Hits@3, and a few-shot large language model reaches 78.73% Hits@1. On sub-action prediction, the same embed","pith_inferences":["The paper leaves implicit that its recording-level split is the right stress test for real deployment: a robot meeting a new household scene must generalize to unseen nodes, so the transductive failure is a substantive finding rather than a mere artifact.","A direct experimental extension is to give unseen test components feature-initialized or inductively computed embeddings; if Hits@k jumps to baseline levels, the gap is explained by node representation rather than by missing relational structure.","The parent/sub-action asymmetry suggests a hybrid design, which the paper mentions only as future work: let a global model choose the goal and a transition-count model choose the next step.","The same disconnected-component structure is likely present in other situational graphs derived from video or sensor streams, so the conclusion may transfer beyond household tasks, though the paper does not test this."],"forward_implications":["On this kind of situational graph, parent-action prediction is solved by an object-aware frequency baseline or a few-shot LLM, while embedding-based KGC models contribute essentially nothing.","For sub-action prediction, the most frequent observed transition conditioned on the current action-object pair is the strongest predictor, and the LLM's sequential reasoning is a liability at this granularity.","The fragmented, hierarchical, temporal structure of situational graphs fails the dense-connectivity assumption of standard KGC benchmarks, so random link masking is an inappropriate evaluation protocol for this setting.","If these results hold, KGC methods for robotics will need inductive or dynamic representations rather than transductive embeddings trained on a single connected graph."],"supporting_citations":[{"why":"Supplies the annotated household-action video dataset from which the situational knowledge graphs are built.","marker":"[8]"},{"why":"Provides the implementation used to train and evaluate the embedding-based link prediction models.","marker":"[1]"},{"why":"TransE is one of the standard embedding models tested; its near-zero performance anchors the failure claim.","marker":"[3]"},{"why":"TransR is another tested embedding model that also fails to beat the baselines.","marker":"[23]"},{"why":"ComplEx is one of the tested embedding models whose low scores support the central claim.","marker":"[37]"},{"why":"DistMult is one of the tested embedding models, with the weakest reported performance.","marker":"[41]"},{"why":"RotatE is the best embedding model on sub-action prediction, bounding what transductive models can do.","marker":"[35]"},{"why":"LiteralE variants test whether adding textual literals rescues the embedding models; they still underperform.","marker":"[18]"},{"why":"The large language model used for few-shot prediction, achieving high parent-action accuracy and low sub-action accuracy.","marker":"[28]"},{"why":"Supplies the few-shot prompting technique used to steer the LLM with intermediate reasoning steps.","marker":"[4]"}],"fun_headline_variants":["KG embeddings near zero on household action prediction","Simple counts beat KG embeddings for household actions","LLM and frequency dominate KG completion on task graphs","Situational graphs expose KG completion failure","Why standard link prediction fails on household task graphs"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The evaluation assumes that embedding models trained on some video recordings can be scored on held-out recordings whose nodes never appeared in training, even though the models have no way to represent those unseen nodes.","fun_headline_variants_meta":{"raw":{"variants":["KG embeddings near zero on household action prediction","Simple counts beat KG embeddings for household actions","LLM and frequency dominate KG completion on task graphs","Situational graphs expose KG completion failure","Why standard link prediction fails on household task graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1132,"prompt_tokens":622,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":366,"completion_tokens_details":{"reasoning_tokens":442}},"tokens_in":366,"tokens_out":510,"duration_ms":6430,"temperature":1.0,"reasoning_tokens":442,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:57:20.364745+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the embedding models with one held-out test component included in the training graph, or initialize unseen test entities with feature-based embeddings, and recompute Hits@k. If scores jump to the baseline range, the reported failure is an artifact of unseen-node evaluation rather than a property of situational graphs; if they stay near zero, the structural mismatch is confirmed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the annotated household-action video dataset from which the situational knowledge graphs are built."},{"cited_title":"PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings","cited_arxiv_id":null,"evidence_quote":"Provides the implementation used to train and evaluate the embedding-based link prediction models."},{"cited_title":"Trans- lating embeddings for modeling multi-relational data","cited_arxiv_id":null,"evidence_quote":"TransE is one of the standard embedding models tested; its near-zero performance anchors the failure claim."},{"cited_title":"Learning entity and relation embed- dings for knowledge graph completion","cited_arxiv_id":null,"evidence_quote":"TransR is another tested embedding model that also fails to beat the baselines."},{"cited_title":"Complex embeddings for simple link prediction, 2016","cited_arxiv_id":null,"evidence_quote":"ComplEx is one of the tested embedding models whose low scores support the central claim."},{"cited_title":"Embedding entities and relations for learning and inference in knowledge bases, 2015","cited_arxiv_id":null,"evidence_quote":"DistMult is one of the tested embedding models, with the weakest reported performance."},{"cited_title":"In- corporating literals into knowledge graph embeddings","cited_arxiv_id":null,"evidence_quote":"LiteralE variants test whether adding textual literals rescues the embedding models; they still underperform."},{"cited_title":"Gpt-4o-mini, 2023","cited_arxiv_id":null,"evidence_quote":"The large language model used for few-shot prediction, achieving high parent-action accuracy and low sub-action accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the few-shot prompting technique used to steer the LLM with intermediate reasoning steps."}],"review_version":1}