{"id":"4eb2bd06-1599-470d-b308-9ce263c1fcde","arxiv_id":"2501.02157","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"PGraphRAG adds neighbor-user reviews to LLM prompts and claims improved personalized generation, but its own ablations show the user's history contributes little beyond item context.","lead":"This paper proposes PGraphRAG, a system that retrieves product reviews from a user's graph neighbors to help language models write personalized text. The authors also release a new sparse-data benchmark, but their own experiments show that neighbor-only context, with no user history, performs just as well as the full method.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The neighbor-only ablation matches the full method, so the reported gains are not shown to come from user-specific personalization.","rationale":"The reader's rejection is supported by a direct contradiction inside the paper: the full PGraphRAG method is not better than its own neighbor-only ablation. This is the most load-bearing issue because the abstract and introduction claim that graph-based retrieval provides 'unique advantages for personalization' and that PGraphRAG 'consistently outperforms' state-of-the-art personalization methods. If the same results can be obtained by retrieving reviews of the same item without attending to the target user, then the method is not personalized in any demonstrated sense; it is item-context RAG on a user-item graph. The reader also correctly notes that the benchmark's construction excludes the very cold-start users the paper says it targets, since Section A.1 requires a shared-item neighbor and filters gold-label selection accordingly. The unresolved leakage question is real but secondary; the ablation alone is sufficient to call the central claim into question. I therefore do not change the reader's verdict: major revision or rejection is needed before the personalization claim can be accepted. I also acknowledge the benchmark resource itself may still be useful, but the scientific claim about PGraphRAG as a personalized method is not supported by the evidence presented.","tokens_in":24221,"tokens_out":5362,"duration_ms":54615,"concrete_test":"Implement an item-only baseline that, for each test instance (user i, item j), retrieves the top-k reviews of item j using the same BM25 or Contriever retriever and the same prompt template, with no user-specific candidate filtering and with the gold review excluded from the retrieved set. Compare its ROUGE-1, ROUGE-L, and METEOR on Tasks 1-8 against the PGraphRAG and PGraphRAG-N values in Tables 9 and 10. If the item-only baseline is statistically indistinguishable from PGraphRAG, the claimed personalization effect is an artifact of item context.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, the user-specific part of the profile defined in Eq. 3 must contribute to the reported gains. The paper's own ablation in Section A.4 shows the opposite. Across Tables 9 and 10, PGraphRAG-N, which retrieves only neighboring users' reviews of the shared item, performs on par with or better than the full PGraphRAG on nearly every task. For example, in Table 9, Hotel Experiences Generation with LLaMA-3.1-8B-Instruct gives ROUGE-1 0.263 for PGraphRAG versus 0.272 for PGraphRAG-N, and with GPT-4o-mini 0.265 versus 0.268; Table 10 shows the same pattern for short-text generation. PGraphRAG-U, restricted to the target user's own history, is consistently worse. This indicates that the improvement over LaMP is driven by item-relevant neighbor content, not by the user-specific profile that Eq. 3 is designed to capture. In addition, Section A.1 only selects users with at least one neighbor-compatible interaction, so the benchmark never evaluates the true cold-start case emphasized in the paper. If a non-personalized item-context retriever reproduces PGraphRAG's numbers, the 'personalized graph-based retrieval' framing collapses into item-context augmentation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes PGraphRAG, a retrieval-augmented generation framework that builds a bipartite user-item graph, defines a user profile as the union of the user's own interactions and other users' interactions on shared items (Eq. 3), retrieves the top-k entries with BM25 or Contriever, and conditions an LLM on the retrieved context. It also introduces a 12-task benchmark constructed from four review corpora, spanning long-text generation, short-text generation, and ordinal rating classification, and reports zero-shot experiments with LLaMA-3.1-8B-Instruct and GPT-4o-mini against No-Retrieval, Random-Retrieval, and LaMP baselines. The paper claims consistent state-of-the-art performance, with average ROUGE-1 gains of 14.8% on long-text and 4.6% on short-text generation, and argues that graph-based neighbor context addresses cold-start and sparse-profile settings.","tokens_in":24384,"tokens_out":6247,"duration_ms":57345,"significance":"If the central claim were established, the paper would contribute a useful public benchmark for personalized text generation under sparse profiles and a modular retrieval framework that goes beyond raw user history. The benchmark spans four domains, three task families, two LLM backbones, and multiple retrievers, and the ablation matrix is more extensive than is typical. The public release of the benchmark and code is also a concrete strength. However, the paper's own evidence undermines the central attribution claim: the neighbor-only ablation performs on par with or better than the full method, so the user-specific component of Eq. (3) is not shown to drive the reported gains; the ordinal-classification tables show the method losing to LaMP on most rating tasks; and the benchmark construction excludes users without shared-item neighbors, so true cold-start performance is not measured. The 'personalized graph-based retrieval' contribution therefore collapses, in the current experiments, into item-context augmentation, and the headline claim of consistent outperformance is not supported by the reported tables.","major_comments":[{"comment":"The neighbor-only ablation undermines the central claim that PGraphRAG's user-specific profile is the source of the gains. In Table 9, PGraphRAG-N achieves ROUGE-1 0.272 for Hotel Experiences Generation with LLaMA-3.1-8B-Instruct and 0.268 with GPT-4o-mini, versus 0.263 and 0.265 for the full PGraphRAG; across nearly all long- and short-text tasks PGraphRAG-N matches or exceeds the full model, while PGraphRAG-U is consistently worse. This shows that the component of Eq. (3) specific to the target user contributes little or nothing, and that the improvement over LaMP is attributable to adding item-relevant neighbor reviews. Since those neighbor reviews are not user-specific, the 'personalized' framing is not supported; a non-personalized item-context retriever reproduces the results.","section":"A.4 (Tables 9 and 10)"},{"comment":"The abstract's claim of consistent outperformance is contradicted by the paper's own rating-task results. For ordinal classification, PGraphRAG outperforms LaMP on only 1 of 4 tasks with LLaMA-3.1-8B-Instruct and 2 of 4 tasks with GPT-4o-mini; for example, on Task 9 User Product Review Ratings, LaMP achieves MAE 0.3132 versus PGraphRAG's 0.3400 with LLaMA and 0.3480 versus 0.3832 with GPT. Even among text-generation tasks, Table 4 shows that on Task 8 with GPT-4o-mini, LaMP is ahead on ROUGE-1, ROUGE-L, and METEOR, and No-Retrieval is ahead on ROUGE-1. The paper's own Section 4.1 acknowledges partial results in the rating tasks, but the abstract, introduction, and conclusion do not qualify the 'consistent' claim.","section":"4.1 (Tables 8 and 18)"},{"comment":"The benchmark does not evaluate the cold-start regime that motivates the paper. The construction requires each selected user to have at least one interaction with an entity also associated with another user, and gold-label selection is performed at the edge level, so users without any neighbor-compatible interactions are excluded from the evaluated test instances. Thus the test set consists only of users who already have at least one shared-item neighbor in the graph, which is not the 'sparse or unavailable' user history setting emphasized in the introduction and limitations. The paper should report the fraction of users excluded by this filter and evaluate a true cold-start split, where the target user has no history or no shared items, with appropriate baselines.","section":"A.1"},{"comment":"No statistical significance testing or confidence intervals are reported for any comparison, despite the abstract's 'significantly outperforms' and the precise average gains of 14.8% and 4.6%. Many headline differences are very small (e.g., Table 4, Task 6 with LLaMA: ROUGE-1 0.127 versus 0.126 for LaMP; Table 3, Task 1 with LLaMA: ROUGE-1 0.178 versus 0.173 for LaMP), and some metric comparisons favor baselines (e.g., Table 3, Task 1 METEOR: No-Retrieval 0.154 versus PGraphRAG 0.151). Without paired significance tests, per-user variance, or confidence intervals, the claimed consistent gains are not established. This is load-bearing for the central claim.","section":"4 (Evaluation)"}],"minor_comments":[{"comment":"The main text states that retrieval uses k=5, but Appendix A.5 reports ablations for k=1,2,4 and Appendix A.7 uses k=4; the default value should be reconciled across the paper.","section":"4 (Ranking and Retrieval) vs A.5/A.7"},{"comment":"The caption says the table shows gains across Tasks 1–7, but the table includes columns for Task 8; the caption or table should be corrected.","section":"A.2 (Table 6)"},{"comment":"The text says the table shows 'review density per product,' but the table reports counts of reviews per product; the wording should be aligned with the actual content.","section":"A.2 (Table 7)"},{"comment":"The phrase 'Muito bom” (Very good)' contains mismatched quotation marks and should be normalized typographically.","section":"A.2"},{"comment":"Table 3 reports Task 1 PGraphRAG ROUGE-1 0.178 for LLaMA-3.1-8B-Instruct on the test set, while the corresponding validation row in Table 16 and the ablation row in Table 9 show 0.173; the paper should clarify which configuration is reported in each table.","section":"Tables 3, 9, and 16"},{"comment":"The quantifiers '∀j∈V, k∈U, k≠i' are placed after the set definition; moving them inside or before the definition would make the notation clearer.","section":"3, Eq. (3)"}],"recommendation":"reject","confidential_remarks":"The stress-test concern lands. The paper's own ablation shows that neighbor-only retrieval matches or exceeds the full method, the rating tables contradict the 'consistent' claim, and the benchmark construction excludes the true cold-start population. These are load-bearing issues that cannot be resolved by local edits. If the authors were to reframe the contribution as graph-based item-context augmentation, remove the cold-start overclaim, and add significance testing, a substantially revised version might be viable, but the current manuscript's central claims are not supported by its evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful benchmark, honest ablations, and a central claim that their own tables contradict. The neighbor-only variant matches the full method on nearly every task, so the 'personalized graph-based retrieval' framing collapses into item-context augmentation. That negative result is the most valuable thing here.\n\nWhat is actually new: a 12-task benchmark for personalized text generation, built from four review datasets with bipartite user-item graphs and very sparse profiles (mean profile size ~1.1). They evaluate zero-shot LLaMA-3.1-8B and GPT-4o-mini with BM25 and Contriever, and run thorough ablations over retrieval scope, k, and retriever. The data release is a real community resource.\n\nWhere the soft spots are, in rough order:\n\n1. Unresolved leakage: Eq. 3 defines the profile as user i's interactions plus neighbors' interactions for the same items. If the gold interaction (i,j) is in the graph, it is inside the profile. The paper never states that the gold review is excluded from retrieval. The ablation numbers suggest the retriever is probably not copying the gold (PGraphRAG-U is mediocre), but the protocol must be explicit. A referee will check this first.\n\n2. The neighbor-only ablation (PGraphRAG-N) matches or beats full PGraphRAG on almost all tasks. The user's own history adds nothing; the gains come from other users' reviews of the item being reviewed. That is item grounding, not user personalization. The authors admit this in A.4 but keep the 'personalization' framing in the abstract.\n\n3. The phrase 'consistently outperforms' is false. In the rating tasks, PGraphRAG loses to LaMP on 3 of 4 tasks with LLaMA and 2 of 4 with GPT. In Task 8, GPT shows negative relative gains on all metrics (Table 6).\n\n4. The benchmark does not test true cold-start. Appendix A.1 requires every sampled interaction to share an item with another user; users without such neighbors are excluded from gold-label selection. So the evaluation is sparse, not unavailable, history.\n\n5. No significance tests. Several reported differences are 0.001, and the best retriever configuration is picked per task on validation, so some gains are likely noise.\n\nRecommendation: this deserves peer review, not desk reject. The benchmark is useful, and the neighbor-only negative result is worth knowing. But the paper needs major revision: state the leakage protocol, add an explicit non-personalized item-context baseline, and rewrite the claims to match the data.","headline":"Useful benchmark, honest ablations, and a central claim that their own tables contradict.","tokens_in":25028,"tokens_out":5232,"would_cite":false,"duration_ms":46704,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that retrieving reviews written by other users about items a user has reviewed improves LLM personalization beyond using the user's own history, with average ROUGE-1 gains of 14.8% on long-text and 4.6% on short-text…","keywords":["personalization","retrieval-augmented generation","knowledge graphs","cold start","sparse user profiles","bipartite graphs","text generation benchmark","large language models"],"falsifier":"Run the framework on the users the benchmark excludes from gold-label evaluation—users whose items have no co-reviewing neighbors—and compare against the user-history-only baseline; if the graph-based gains disappear or shrink sharply, the benefit comes from item-level context rather than personalized voice. Equivalently, replace neighbor reviews with non-personal item descriptions and observe whether the ROUGE gains persist.","tokens_in":23950,"feed_emoji":"🤖","tokens_out":6316,"duration_ms":53814,"temperature":0.7,"pith_summary":"The paper sets out to show that structured graph context — specifically, reviews written by other users about the same items a target user has reviewed — can personalize large language model outputs even when the target user's own history is nearly empty. To this end it introduces PGraphRAG, a retrieval-augmented generation framework that builds a bipartite user–item graph, defines each user's profile as their own interactions plus neighbor interactions on shared items, and retrieves the most relevant entries to condition generation. It also releases a twelve-task benchmark spanning long-form generation, short-form generation, and rating classification, constructed from four real-world review datasets. Across these tasks, PGraphRAG reports consistent gains over a user-history-only baseline, with the largest gains on hotel review generation. A sympathetic reader would take the paper's contribution to be evidence that neighbor information from a graph can substitute for missing personal history in cold-start settings.","feed_headline":"Neighbor reviews improve LLM personalization","feed_subtitle":"Graph-based retrieval pulls other users' reviews of shared items, beating history-only baselines on sparse-profile text generation.","key_machinery":"The central machinery is the user-centric bipartite graph $G = (U, V, E)$, where $U$ is the set of users, $V$ is the set of items, and an edge $(i,j)$ is a review with text, title, and rating. The load-bearing identity is Eq. (3), the profile definition $P_i = \\{(i,j) \\in E\\} \\cup \\{(k,j) \\in E \\mid (i,j) \\in E\\}$, which extends a user's own sparse history with neighboring users' reviews of the same items. The retrieval pipeline of query function, top-$k$ graph retrieval, and prompt construction turns this profile into a context-augmented prompt $\\tilde{x} = \\phi_p(x, R(\\phi_q(x), G, k))$ used for inference or fine-tuning. Ablations show that the neighbor-only variant (PGraphRAG-N) achieves scores on par with the full method, while the user-only variant (PGraphRAG-U) falls behind, indicating that the graph-neighbor expansion is what carries the reported gains.","core_discovery":"On the paper's own terms, the central discovery is that retrieving context from a user-centric bipartite graph—rather than only from the user's own past texts—yields better personalized generation in sparse-profile regimes. The profile of user $i$ is defined as $P_i = \\{(i,j) \\in E\\} \\cup \\{(k,j) \\in E \\mid (i,j) \\in E\\}$, i.e., the user's own reviews plus all reviews by other users $k$ of the same items $j$. The framework then applies a query function, a graph-based retriever (BM25 or Contriever) that selects the top-$k$ entries from this profile, and a prompt constructor that inserts the retrieved entries into a template; the resulting prompt conditions the LLM on both the input and the retrieved context. The paper reports that this approach outperforms the LaMP baseline on nearly all of the twelve benchmark tasks, and that a neighbor-only variant (excluding the target user's own reviews) performs on par with the full method—evidence, in the authors' interpretation, that community signals across shared items carry much of the personalization value when direct history is thin.","pith_inferences":["If neighbor reviews carry item facts rather than user voice, the method is better described as collaborative item-context augmentation, and the 'personalization' framing may overstate what is learned about the user; a controlled test that swaps user identities on the same retrieved reviews would distinguish the two interpretations.","The strong performance of the neighbor-only variant in review-generation tasks suggests that genre conventions and item attributes dominate what looks like personal style; applying the method to tasks with genuinely idiosyncratic outputs, such as email tone or creative writing, would test the personalization claim more sharply.","The benchmark's filtering rule—requiring every evaluated user to share at least one item with another user—means the truly isolated cold-start population is absent from the gold labels; an evaluation on that excluded population would clarify the real-world deployment value.","Because the method needs no parametric user embeddings and retrieval is scoped to a localized subgraph, it could be adapted to privacy-preserving settings where user histories cannot leave the device, provided the shared-item graph can be constructed."],"forward_implications":["Neighbor-only retrieval, which excludes the target user's own history, matches the full method on most tasks, so a user with a single review can still receive personalized generation through the community of users who reviewed the same items.","The ranking of retrieved entries matters: randomly sampling the same number of profile entries instead of ranking them drops ROUGE-1 by 2.29% on long-text and 3.18% on short-text tasks relative to ranked retrieval.","Performance is robust to the choice of retriever, with BM25 and Contriever yielding comparable results, which suggests the gains come from the graph-based profile structure and prompt construction rather than from a specific retrieval model.","Increasing retrieval depth from $k=1$ to $k=4$ generally improves long-text generation with diminishing returns, but the benefit is capped by profile sparsity because many users have fewer than four qualifying interactions.","The released benchmark includes users with average profile sizes around 1.05–1.14 reviews, simulating cold-start conditions that prior personalized-generation benchmarks filtered out."],"supporting_citations":[{"why":"Defines the LaMP benchmark and the user-history-retrieval baseline that PGraphRAG is compared against on every task.","marker":"Salemi et al. 2024b"},{"why":"Introduces LongLaMP, the prior benchmark for personalized long-form text generation that this work extends to graph-based, sparse-profile settings.","marker":"Kumar et al. 2024"},{"why":"Supplies the Amazon Reviews 2023 corpus, the data source for the user-product review graph used in Tasks 1, 5, and 9.","marker":"Hou et al. 2024"},{"why":"Provides the BM25 sparse retriever used as one of the two retrieval backends inside the graph-based retrieval module.","marker":"Robertson and Zaragoza 2009"},{"why":"Provides Contriever, the dense unsupervised retriever used as the alternative backend in the retriever ablation.","marker":"Lei et al. 2023"},{"why":"Contributes the B2W-Reviews corpus that forms the multilingual Brazilian-Portuguese graph for Tasks 4, 8, and 12.","marker":"Real et al. 2019"},{"why":"Supplies the Hotel Reviews dataset used to build the user-hotel graph for Tasks 2, 6, and 10.","marker":"Datafiniti 2017"},{"why":"Supplies the Grammar and Online Product Reviews corpus used for the stylized feedback tasks 3, 7, and 11.","marker":"Datafiniti 2018"}],"fun_headline_variants":["Graph retrieval boosts LLM personalization","Neighbor reviews key to LLM personalization","Cold-start fix: graph-based context for LLMs","LLMs get personal via user knowledge graphs","Community context improves LLM personalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that other people's reviews of the same product or hotel reflect the target user's own voice and preferences, rather than merely supplying facts about the item.","fun_headline_variants_meta":{"raw":{"variants":["Graph retrieval boosts LLM personalization","Neighbor reviews key to LLM personalization","Cold-start fix: graph-based context for LLMs","LLMs get personal via user knowledge graphs","Community context improves LLM personalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1452,"prompt_tokens":942,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":443}},"tokens_in":558,"tokens_out":510,"duration_ms":4638,"temperature":1.0,"reasoning_tokens":443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:56.147743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on the users the benchmark excludes from gold-label evaluation—users whose items have no co-reviewing neighbors—and compare against the user-history-only baseline; if the graph-based gains disappear or shrink sharply, the benefit comes from item-level context rather than personalized voice. Equivalently, replace neighbor reviews with non-personal item descriptions and observe whether the ROUGE gains persist.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the B2W-Reviews corpus that forms the multilingual Brazilian-Portuguese graph for Tasks 4, 8, and 12."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Hotel Reviews dataset used to build the user-hotel graph for Tasks 2, 6, and 10."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Grammar and Online Product Reviews corpus used for the stylized feedback tasks 3, 7, and 11."}],"review_version":1}