{"id":"f6baa16a-eced-43e1-b918-a376c0031e3d","arxiv_id":"2507.13910","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"PARK personalizes academic search by embedding a citation-derived knowledge graph into the same vector space as a neural retrieval model, beating baselines in three of four domains.","lead":"PARK is a two-stage retrieval system that combines a neural language model with knowledge graph embeddings built from citation data to personalize academic search. In tests on four field-specific datasets it outperformed existing baselines in three of them, by up to 10% in mean average precision.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim depends on whether the knowledge graph includes test-period query papers; if it does, 'Cited' edges encode the benchmark relevance labels directly, making the Table 1 gains a leakage artifact.","rationale":"The reader's weakest assumption identifies the same load-bearing concern I would flag: the knowledge graph's temporal construction is never explicitly restricted to pre-cutoff documents, and the 'Cited' relation can directly encode test-period relevance labels if it is not. This is not a disagreement with the modeling choice; TransE/TransH on citation graphs are reasonable, and the two-stage pipeline is coherent. The issue is that the evaluation protocol is underspecified at exactly the point where the strongest empirical claim lives. The paper does say the dataset is split chronologically 'to prevent data leakage' (Section 4.1), which suggests the authors intended a clean split, but that sentence only describes the training and validation sets for the retrieval model, not the KG construction in Section 3.2. Since the public code is available, the ambiguity is resolvable. If the KG includes test-period query papers, the user embedding for a query author is trained with explicit edges to the documents that the benchmark says should be retrieved; because document embeddings are fixed, those edges directly position the user embedding near the relevant documents. That would make the measured MAP improvements a form of label leakage rather than evidence about user modeling. If the KG is properly filtered, I see no comparable flaw in the central argument; the remaining issues of lambda tuning details, significance testing, and externally reported baselines are secondary and addressable. I therefore do not move the reader's verdict: it stays conditional pending the concrete check, with the direction of adjustment depending on what the code shows.","tokens_in":15478,"tokens_out":7682,"duration_ms":93441,"concrete_test":"Inspect the public repository's KG construction script for the cutoff filter. Concretely, rebuild the KG using only documents with publication date before each domain's cutoff (2016 for Political Science, 2017 for Computer Science and Physics, 2019 for Psychology), drop all relations incident to post-cutoff documents including test query papers, retrain the TransE/TransH user embeddings with the same hyperparameters, re-tune lambda on the validation split, and recompute Table 1. If PARK-E/H no longer achieve a statistically significant MAP@100 advantage over the second-best baseline in Political Science, Psychology, and Physics, the original gains depend on test-period citation edges and the central claim should be rejected. If the code already contains such a filter, the concern is resolved and the reader's conditional verdict can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result is Table 1: PARK-E/H beat the second-best model by roughly 9-10% MAP@100 in Political Science, Psychology, and Physics. For that result to support the claim that knowledge-graph user models improve personalized academic search, the knowledge graph used to build user embeddings must not contain the test-period query paper. The paper never states this. Section 4.1 gives a chronological split only for the training sets of the retrieval model; Sections 3.2 and 3.3 describe the KG and user scoring without any temporal filter on the citation graph. If the KG is built from the full collection, then for a test query paper p (published after the cutoff) and its author u, the KG contains (u, Wrote, p) and, via the Cited relation, (u, Cited, d) for every d cited by p. In the benchmark, those d are exactly part of the relevance labels for the query. Because Section 3.2 fixes document-node embeddings to the MiniLM encoder, each such edge pulls the user embedding directly toward the embedding of a relevant document; the user-similarity term in the convex combination of Section 4.2 then boosts exactly those documents. The same leak may also inflate the Self Citation baseline, but the 8-10% margin of PARK over that baseline is not evidence against leakage. The missing sentence is small but load-bearing: if code inspection shows post-cutoff documents, or their Cited edges, are present in the KG, the measured improvements are an artifact of label injection and the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PARK, a personalized academic search pipeline in which a BM25 first stage and a MiniLM dense retriever are augmented by a user model built from knowledge-graph embeddings of the academic citation graph. The citation graph is converted into a knowledge graph with user, document, venue, and affiliation nodes and relations such as Wrote, Cited, In Venue, Affiliated, and Co-author; TransE (PARK-E) and TransH (PARK-H) are trained to embed this graph into the same space as the dense retriever, with document embeddings fixed. Documents are finally ranked by a convex combination of BM25, dense similarity, and user-similarity scores. Experiments on a four-domain benchmark (Computer Science, Political Science, Psychology, Physics) claim state-of-the-art MAP@100 in three domains, with about 9--10% relative improvement over the second-best model, plus an ablation on node types.","tokens_in":15758,"tokens_out":3189,"duration_ms":37591,"significance":"If the results are valid, PARK offers a practical integration of knowledge-graph embeddings with neural retrieval for personalized academic search, and the public code and benchmark would support reproducibility. The two-stage design, where document embeddings are shared between the dense retriever and the KG embedding model, is a sensible way to combine lexical, semantic, and structural signals, and the ablation provides useful evidence about which KG nodes contribute. However, the central claim is currently threatened by an unstated temporal split in the knowledge-graph construction, and several experimental details are missing; the significance of the claimed gains cannot be assessed until that issue is resolved.","major_comments":[{"comment":"The paper does not state whether the knowledge graph used for user embeddings is restricted to documents published before the chronological cutoff. This is load-bearing because of the Cited relation: for an author u who authored a test-period query paper p, the edge (u, Cited, d) exists for every document d cited by p, and those d are exactly the relevance labels in the benchmark (the references of the query paper). Since document-node embeddings are fixed to the MiniLM encoder, such edges would pull the user embedding directly toward relevant documents, making the user-similarity score trivially predictive and the Table 1 gains a leakage artifact. The sentence in §4.1 about preventing data leakage refers only to the training sets of the retrieval model; the KG construction is not described as respecting the same split. The authors must specify precisely which papers, authors, and citation edges are included in the KG, and ideally provide a check (e.g., the number of test-period query papers excluded) or a controlled experiment where the KG is built only from pre-cutoff documents.","section":"§3.2, §4.1"},{"comment":"The combination weights λ1, λ2, and λ3 are said to be optimized on the validation set, but their values are never reported. Since all baselines are combined with the same convex weighting, the final ranking can be sensitive to these weights, and the reader cannot judge whether the reported margins are due to the user model or to weight tuning. Report the selected λ values for each domain and model, or show the sensitivity of MAP@100 to these weights.","section":"§4.2, Eq. (1)"},{"comment":"The claim of statistical significance is not supported by any description of the test used. The asterisk is defined as 'a statistically significant difference over the second-best-performing model,' but the paper does not state whether a paired test (e.g., paired t-test or Wilcoxon) was used, over how many queries, or how multiple comparisons were handled. This is necessary both for the headline improvements and for the Computer Science case, where PARK-H is not marked as significant relative to the second-best model. Please specify the significance test, the number of query-user pairs, and the threshold.","section":"§4.3, Table 1"}],"minor_comments":[{"comment":"The user similarity score is described only as measuring similarity between the query writer and 'the authors of the documents being scored'; it is not clear whether the score is a maximum over the document's authors, a mean, or something else. A precise formula is needed to reproduce the method.","section":"§3.3"},{"comment":"In the triplet margin loss equation, the first norm uses a subscript p while the second uses subscript 2; this appears to be a typo and should be consistently ||·||₂.","section":"§4.1, Eq. (1)"},{"comment":"The PageRank baseline is applied to a citation graph, but the paper does not specify whether this graph is the full collection or only the training-period documents. If the test-period citations are included, PageRank could also be influenced by the same leakage path, weakening its comparability as a baseline.","section":"§4.2"},{"comment":"In the Computer Science column, PARK-H has 0.230 MAP@100 while POP has 0.238; the text states PARK-H 'falls short by 3%' but the relative difference is about 3.4%, and the direction of the comparison should be stated more precisely in the text.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The leakage question is the central issue. The paper's own code is public, so the authors can quickly verify whether the KG contains test-period query papers or their cited edges. If it does, the main experiments are invalid and the paper would need to be rerun with a proper temporal split; if it does not, the authors should make that fact explicit in both §3.2 and §4.1. I lean toward major_revision rather than reject because the flaw is a missing specification that is likely fixable, but the authors must provide concrete verification rather than a mere assertion. Also check whether the baseline results for CTRLIt and CrossEncRA are comparable: the paper reports their numbers from the original papers, but those models may use different retrieval pipelines or user profiles, which would make direct comparisons in Table 1 misleading."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read PARK last night. The core idea is genuinely new: align TransE/TransH KG embeddings of the citation graph into the same space as a frozen MiniLM document encoder, then use the resulting user embeddings to re-rank. That combination has not been done for academic search, and the two-stage design is clean: no retraining of the LM, and you can pre-index document vectors. Credit where due: the paper is well written, the ablation on node types is informative, and they ship code.\n\nThe soft spot is the one the stress test flagged. The KG construction in Section 3.2 never states whether it includes test-period papers. In this benchmark, relevance for a query paper is exactly the papers it cites (plus BM25 top-100). If the KG contains the query paper and its Cited edges, the user embedding is pulled directly toward the labelled relevant documents, and the 9-10% MAP gains in three domains are a label-injection artifact. The text gives a chronological split only for the retrieval training sets, not for the KG. That missing sentence is load-bearing. I don't think this is a fabricated flaw; the paper really doesn't say.\n\nOther issues are minor: the lambda weights are tuned on validation but not reported, the statistical test is only an asterisk, and two baselines rely on numbers from their papers. None of these are fatal if the leakage question resolves.\n\nRecommendation: send to peer review. The idea is worth refereeing, the code is public, and the leakage concern is checkable. If a referee opens the repo and sees post-cutoff documents or their Cited edges in the KG, the empirical claim falls; if not, this is a solid incremental contribution. I would not cite it before that check.","headline":"Clean two-stage personalization pipeline with a real leakage question that code inspection must resolve.","tokens_in":16284,"tokens_out":2222,"would_cite":false,"duration_ms":24407,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowledge-graph embeddings of citation networks improve personalized academic search in three of four domains tested.","keywords":["Personalized academic search","Knowledge graph embeddings","TransE","TransH","Citation graph","User modeling","Dense retrieval","Neural information retrieval"],"falsifier":"Train and evaluate PARK-H twice: once with the knowledge graph restricted to pre-cutoff documents and once with test-period documents, and their citation edges, included; then compare MAP@100 on the same test queries. If the restricted-graph run loses most of the reported improvement, the gains come from encoding the answer key into the user profile rather than from genuine personalization.","tokens_in":15243,"feed_emoji":"🎓","tokens_out":4631,"duration_ms":49091,"temperature":0.7,"pith_summary":"The paper aims to show that user models built from knowledge-graph embeddings of an academic citation graph can substantially improve personalized academic search, going beyond what lexical, dense, and simpler user-model baselines achieve. The proposed system, PARK, first trains a bi-encoder neural retriever and then embeds a citation-derived knowledge graph into the same vector space using TransE or TransH, with paper embeddings pinned to the retriever's. At query time, the query author's embedding is compared with each candidate paper's authors, and this user-similarity score is fused with BM25 and dense scores. On four domain-specific benchmarks, the method achieves state-of-the-art MAP@100 in three domains with up to 10% relative improvement over the second-best baseline. The central interest is that structural citation relationships can be converted into a continuously updated user profile without needing search-session logs.","feed_headline":"Citation-graph embeddings lift academic search results by up to 10%","feed_subtitle":"User profiles built from citation structure outperform dense-retrieval baselines in most academic fields tested.","key_machinery":"The central object is the academic knowledge graph built from the citation graph, with four entity types (user, document, venue, affiliation) and five relations (Wrote, Cited, In Venue, Affiliated, Co-author). It is embedded with TransE or TransH into the same space as the neural retriever's document embeddings, holding document embeddings fixed. The user embedding of the query author is then compared with embeddings of candidate paper authors to produce a user similarity score; the final ranking is a convex combination of BM25, dense similarity, and user similarity. The Cited relation, which connects a user to every document cited by their papers, is the mechanism that propagates a researcher's reading and reference habits into their profile.","core_discovery":"The paper claims that PARK-E and PARK-H, user models derived from translational knowledge-graph embeddings of the academic citation graph, significantly improve personalized academic search effectiveness. Aligning the knowledge-graph embedding space with the dense retriever's document embeddings, while keeping document embeddings fixed, places users and documents in a shared semantic space where a user-author cosine similarity becomes a meaningful relevance signal. Experiments across four domains show PARK-H improves MAP@100 over the second-best method by 9% in Political Science and Physics and 10% in Psychology, with statistically significant gains; in Computer Science the popularity-based POP baseline remains slightly ahead. The ablations indicate that user-node relations (Wrote, Cited, Co-author) drive most of the gain, affiliation adds institutional context, and venue contributes little.","pith_inferences":["The reported gains may partly reflect the benchmark's construction, since relevance labels are the references of the query paper and the Cited relation is exactly that signal; a cleaner test would hold out the query paper itself from the graph.","The method effectively performs a form of collaborative filtering where transitive citation proximity replaces rating similarity; one could test whether simpler graph-walk or PageRank-style user scores match the knowledge-graph embeddings' performance.","Because document nodes are frozen, the knowledge-graph embedding quality depends entirely on the language model's document representations; relaxing that constraint might improve or distort user embeddings, an easily testable variant."],"forward_implications":["If the central claim holds, academic search systems can build effective user profiles from the citation graph alone, without needing search-session logs or explicit user feedback.","The two-stage training means any dense retriever can be paired with a knowledge-graph-based user model by simply fixing its document embeddings, avoiding joint retraining.","The approach transfers across domains (Computer Science, Physics, Political Science, Psychology), suggesting the citation-structure signal is general rather than domain-specific.","The Computer Science exception indicates that in fields where citation counts are highly informative, popularity features may dominate; combining knowledge-graph user models with popularity features is a natural next step.","The ablation result, that affiliation helps while venue does not, gives a concrete recipe for which graph nodes to include when building such user models."],"supporting_citations":[{"why":"Provides the multi-domain personalized search benchmark, datasets, and relevance labels that all experiments are evaluated on.","marker":"[3]"},{"why":"Introduces TransE, the translational embedding algorithm used by PARK-E.","marker":"[36]"},{"why":"Introduces TransH, the translational embedding algorithm used by PARK-H, designed to handle complex relations.","marker":"[37]"},{"why":"MiniLM is the distilled transformer used as the neural dense retriever whose document embeddings anchor the knowledge-graph space.","marker":"[43]"},{"why":"BM25 is the lexical first-stage ranker that supplies candidate documents and one of the three fused scores.","marker":"[39]"},{"why":"Defines the triplet margin loss used to train the neural retrieval model.","marker":"[7]"},{"why":"States the Closed World Assumption under which the knowledge graph is embedded.","marker":"[40]"}],"fun_headline_variants":["Knowledge-graph user models boost academic search by 10%","PARK: citation graph to knowledge graph lifts search by 10%","Embedding citation graphs puts users and papers in one space","Up to 10% better MAP from knowledge graph user models","Graph embeddings align user interests with papers for better search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the knowledge graph being built from documents published before the evaluation cutoff, so that the Cited relation does not already include the references of the test-period query papers, which are exactly the relevance labels used for scoring.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge-graph user models boost academic search by 10%","PARK: citation graph to knowledge graph lifts search by 10%","Embedding citation graphs puts users and papers in one space","Up to 10% better MAP from knowledge graph user models","Graph embeddings align user interests with papers for better search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000591,"raw_usage":{"total_tokens":2754,"prompt_tokens":910,"completion_tokens":1844,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1759}},"tokens_in":526,"tokens_out":1844,"duration_ms":15416,"temperature":1.0,"reasoning_tokens":1759,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:13:59.377932+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train and evaluate PARK-H twice: once with the knowledge graph restricted to pre-cutoff documents and once with test-period documents, and their citation edges, included; then compare MAP@100 on the same test queries. If the restricted-graph run loses most of the reported improvement, the gains come from encoding the answer key into the user profile rather than from genuine personalization.","supporting_citations":[{"cited_title":"Bordes, N","cited_arxiv_id":null,"evidence_quote":"Introduces TransE, the translational embedding algorithm used by PARK-E."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MiniLM is the distilled transformer used as the neural dense retriever whose document embeddings anchor the knowledge-graph space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BM25 is the lexical first-stage ranker that supplies candidate documents and one of the three fused scores."},{"cited_title":"Balntas, E","cited_arxiv_id":null,"evidence_quote":"Defines the triplet margin loss used to train the neural retrieval model."},{"cited_title":"Reiter, On closed world data bases, in: B","cited_arxiv_id":null,"evidence_quote":"States the Closed World Assumption under which the knowledge graph is embedded."}],"review_version":1}