{"id":"60d19bad-cba4-4948-8b54-04fb9913f5ce","arxiv_id":"2505.00886","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A content-based recommender that builds user profiles from LLM-written short-term and long-term summary texts, fuses them with attention, and claims modest ranking gains plus built-in textual explanations.","lead":"This paper uses large language models to write two separate text summaries of each user's interaction history, one for recent tastes and one for long-term tastes, then fuses these summaries into a single user profile for content-based recommendations. It argues these text summaries and the fusion weights can tell users why a suggestion is being made.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM profiles may be generated from the complete user history, including validation and test interactions, so the reported ranking gains could reflect leakage rather than genuine temporal modeling.","rationale":"The central claim of the paper is the measured improvement in ranking accuracy from the proposed LLM-based temporal user profiles. The weakest point in the argument is the ambiguity about what data the LLM sees when generating user summaries. Section 3.1 repeatedly says 'complete interaction history' and 'entire user interaction history' is fed to the LLM, while Section 4.3 defines a temporal split but never connects that split to profile construction. This is exactly the kind of hidden data-flow issue that can invalidate an experimental comparison: if GPT-4o-mini sees test items, the profile embeddings are computed with knowledge of the labels they are later asked to rank. The reported 17% Recall@10 gain and 14% NDCG@10 gain over Centric are large enough to be entirely explained by such leakage, especially because the test portion, the most recent 20% of interactions, is precisely what the short-term profile is designed to emphasize. A secondary issue is that the explainability claim is stronger than what is demonstrated: Figure 2's part (C), the actual user-facing explanation merging textual profiles with attention weights, is explicitly labeled a hypothetical future extension. However, the accuracy claim is the primary basis for the paper's contribution, and the leakage question is the most consequential. The reader's verdict of CONDITIONAL is appropriate: the paper should be accepted only if the code confirms that summaries are generated from the training prefix and the results are reproduced under that condition. My stress-test pass does not change that verdict, so I mark it UNCHANGED.","tokens_in":12162,"tokens_out":3543,"duration_ms":39005,"concrete_test":"Inspect the provided GitHub repository's profile-generation script to determine whether the H_u passed to GPT-4o-mini is the full chronological history or only the first 60% (training portion) of each user's interactions. Then rerun the Movies&TV experiment with summaries generated only from the training prefix, keeping all other hyperparameters and baselines unchanged. If the proposed method's Recall@10 and NDCG@10 gains over Centric drop from 17%/14% to near zero, or fall below Temp-Fusion, leakage is the cause and the central accuracy claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines H_u as the user's complete interaction history and Eqs. (2)-(3) feed this H_u in full to the LLM for both short-term and long-term summary generation. Section 4.3 then splits each user's history chronologically into training (60%), validation (20%), and test (20%), but the paper never states that profile generation is restricted to the training prefix. If the implementation follows the text literally, GPT-4o-mini summarizes the entire history, including held-out validation and test interactions. On Movies&TV, where the mean profile has 11.79 interactions and the test split is the most recent 20%, the short-term summary can directly mention the user's most recent held-out items; after BERT encoding, those item embeddings will match the user profile, inflating Recall@10 by the reported 17% and NDCG@10 by 14% relative to the Centric baseline. This would be train/test contamination in the user representation, not evidence that time-aware text-based profiling improves ranking. The same concern applies to the Video Games results, although the smaller profiles make the effect weaker. If the code happens to restrict summaries to training interactions, the concern is resolved, but the manuscript as written does not say so.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a content-based recommendation framework that builds temporal user profiles by prompting an LLM (GPT-4o-mini) to produce separate short-term and long-term natural-language summaries of each user's interaction history, encoding these summaries with SBERT, and fusing the two embeddings through a learned attention mechanism before an MLP scores user-item pairs. The authors evaluate the method on Amazon Movies&TV and Video Games data against a centric embedding-average baseline, a non-LLM temporal fusion baseline, popularity ranking, and matrix factorization, and they report Recall@K and NDCG@K improvements, with a focus on the Movies&TV domain. They also claim intrinsic explainability through the readable textual summaries and the short-term/long-term attention weights, and they include an ablation study on the Movies&TV data that isolates the contribution of the temporal split, LLM text, attention fusion, and the MLP scoring function.","tokens_in":12413,"tokens_out":2530,"duration_ms":26985,"significance":"If the reported gains are valid, the paper offers a practical and easy-to-communicate way of making content-based user profiles both temporal and explainable: the textual short-term and long-term profiles are human-readable by construction, and the attention weights provide a simple numerical decomposition of the user representation. The authors provide code, data, prompts, and results on GitHub, and they run ablations over the main architectural choices, which supports reproducibility. The central empirical claim, however, currently rests on an ambiguous data-handling step that potentially leaks test information into the profiles, and the conclusion overstates the comparison against the strongest baseline. These issues need to be resolved before the reported accuracy benefits can be taken as established.","major_comments":[{"comment":"The manuscript never states that the LLM summaries are generated from the training portion of each user's history only. Equations (2)-(3) define NL^short and NL^long as functions of H_u, and Section 3.1 explicitly says the LLM processes the entire interaction history. Section 4.3 then splits each history chronologically into 60% training, 20% validation, and 20% test, but the profiling step is not restricted to the training prefix. If the implementation follows the text literally, the GPT-4o-mini summaries can mention held-out validation and test interactions, and after SBERT encoding those items' embeddings will match the profile, inflating Recall@K and NDCG@K. On Movies&TV, where the mean profile has 11.79 interactions and the test split is the most recent 20%, the short-term summary could directly encode the very items being ranked, which would make the reported 17% Recall@10 and 14% NDCG@10 gains over Centric an artifact of leakage rather than evidence for temporal text-based profiling. The authors must either state explicitly that summaries are generated from the training prefix only and describe how the split is applied before LLM prompting, or modify the methodology accordingly.","section":"Section 3.1 and Section 4.3 (Eqs. 2-3)"},{"comment":"The conclusion states that 'time-aware, text-based user representations yield up to 17% higher Recall@10 and 14% higher NDCG@10 compared to a standard centric baseline,' but the paper's broader framing in the abstract and Section 4.2 claims superiority 'over multiple baselines.' On Video Games, Table 3 shows Temp-Fusion outperforms the proposed method on Recall@10 (0.0693 vs. 0.0665), NDCG@10 (0.0589 vs. 0.0547), and NDCG@20 (0.0712 vs. 0.0683), while the proposed method wins only on Recall@20. The asterisk in Table 3 is computed only against Centric, so no statistical significance is reported against the strongest baseline. The conclusion and abstract should be reworded to acknowledge that the proposed method does not consistently beat Temp-Fusion, and the paper should report significance tests or variance estimates for the comparisons that support the central claim.","section":"Table 3 and Section 6 (Conclusion)"},{"comment":"The discussion of the Video Games results is internally inconsistent with the data in Table 3. The text says Temp-Fusion 'narrowly edges out our method on Recall@10, NDCG@10, and NDCG@20,' which is accurate, but the same paragraph begins with 'our approach still achieves the highest Recall@20' and then attributes the mixed result only to sparse histories and stable interests. If Temp-Fusion beats the method on three of four metrics, the paper should either report this as a limitation of the LLM-based profiling in small-profile domains or provide an analysis of why the LLM text representation underperforms simple embedding fusion there. As written, the narrative downplays a result that directly qualifies the paper's central claim.","section":"Section 4.5, Video Games discussion"}],"minor_comments":[{"comment":"The sentence beginning 'Similarly, [2] interpretability by summarizing...' is missing a verb and is grammatically incomplete; it should read something like 'Similarly, [2] achieves interpretability by summarizing...'.","section":"Section 2, Related Work"},{"comment":"Even if the intended implementation restricts summaries to the training prefix, the text 'the complete interaction history' and 'keeping the full interaction history intact' is now misleading given the temporal split in Section 4.3. Please rephrase to define H_u as the training-prefix history used for profiling and state this before Eqs. (2)-(3).","section":"Section 3.1, User Profile Creation"},{"comment":"The paper reports a single run per configuration and does not provide standard deviations, confidence intervals, or per-seed results. The claim of statistical significance against Centric would be more convincing if the test procedure (e.g., number of seeds, paired test) were described, and if significance were also reported for the Temp-Fusion comparison.","section":"Table 3 and experimental setup"},{"comment":"Figure 2(C) is described as a 'hypothetical extension' and 'future scenario,' but the surrounding text also says the attention weights 'can be surfaced in a real system' and calls the framework 'explainable by construction.' Please clarify exactly which explainability outputs are implemented and evaluated in the current experiments and which are only proposed for future work.","section":"Figure 2 and Section 4.5"},{"comment":"The numerical gains cited in the conclusion (17% Recall@10, 14% NDCG@10) are the Movies&TV gains over Centric. The Video Games gains are much smaller (3-10%). Please add the dataset qualifier in the conclusion as it appears in the abstract, or the claim will be read as applying to both domains.","section":"Section 6, Conclusion"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper's architecture is a sensible new combination of established components: LLM-generated short-term and long-term text summaries, SBERT encoding, attention fusion, and an MLP scorer. The ablation on Movies&TV supports each piece earning its keep, and the authors are honest that Temp-Fusion beats them on several Video Games metrics. Second, the paper has an unresolved leakage ambiguity. Section 3.1 says the LLM receives the complete interaction history H_u; Section 4.3 splits histories into train/validation/test, but the paper never states that H_u is truncated to the training prefix. If it isn't, the short-term summary can mention held-out test items, which would manufacture the reported 17% Recall@10 gain. The code is on GitHub, so a referee can check, but as written the manuscript does not rule out leakage.\n\nWhat's new: the specific combination is not in prior work, and the paper ships code, prompts, and data-processing details. That is a real plus. The explainability story (textual profiles plus attention weights) is a reasonable interpretive claim, not a fully evaluated user-facing feature; the authors themselves say a user study is future work.\n\nSoft spots, in order of severity. The leakage question is load-bearing; the central empirical claim collapses if summaries include test interactions. The paper also reports no variance or significance testing against Temp-Fusion, the closest baseline; on Video Games Temp-Fusion wins Recall@10, NDCG@10, and NDCG@20, which undercuts the abstract's 'boosting recommendation accuracy over multiple baselines' phrasing. The significance asterisks in Table 3 only compare against Centric, not against Temp-Fusion. Finally, 'explainable by construction' is asserted, not measured; the attention-weight explanation is plausible but unvalidated.\n\nThe citation pattern looks fair; no obvious missing work.\n\nBottom line: this deserves a serious referee, not a desk reject, because the ambiguity is checkable and the architecture is reasonable. If the code confirms training-prefix summarization, the paper is a moderate contribution to content-based recommendation. If not, the empirical core is invalid. Either way, the paper should not be published as-is; it needs a clear statement about what the LLM saw and ideally error bars. Who it's for: researchers working on LLM-based user modeling and recommender evaluation. A workshop reviewer would get value from it, mainly as a cautionary example about temporal leakage.","headline":"A reasonable LLM-based temporal profiling architecture whose empirical claim is jeopardized by an unresolved question about whether the LLM sees test interactions during summary generation.","tokens_in":12921,"tokens_out":2894,"would_cite":false,"duration_ms":29141,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using LLM-generated short- and long-term text profiles of a user's history improves recommendation ranking over averaged-embedding profiles and provides explanations by construction.","keywords":["Explainable user modeling","Content-based recommendation","Large language models","Temporal user profiling","Short-term and long-term preferences","Attention fusion","Text-based user representation"],"falsifier":"Regenerate the short-term and long-term summaries using only the earliest 60% of each user's chronological history (the training split) and re-run the same ranking evaluation; if the improvement over the centric baseline drops or disappears, the original results were inflated by seeing held-out items.","tokens_in":11949,"feed_emoji":"📝","tokens_out":8351,"duration_ms":72751,"temperature":0.7,"pith_summary":"The paper proposes a content-based recommender in which each user's chronological interaction history is fed to an LLM twice with different prompts, producing a short-term profile emphasizing recent behavior and a long-term profile of stable tastes. These text profiles are encoded into embeddings, fused by a per-user attention weight, and scored against item embeddings with an MLP. On the Movies&TV dataset the method beats the standard centric averaging baseline by up to 17% in Recall@10 and 14% in NDCG@10; on the sparser Video Games dataset the gains are smaller and the simpler temporal fusion baseline edges it out on several metrics. Because the original text summaries and the learned attention weights are human-readable, the model can explain its recommendations without a separate explanation module.","feed_headline":"LLM text profiles lift recommendation recall up to 17 percent","feed_subtitle":"Splitting user histories into short- and long-term summaries also makes recommendations explainable","key_machinery":"The core mechanism is two-pass LLM summarization of the same interaction history with distinct prompts, producing textual profiles $\\mathrm{NL}_{\\mathrm{short}}^u$ and $\\mathrm{NL}_{\\mathrm{long}}^u$; the short-term prompt biases toward recent interactions while the long-term prompt extracts persistent patterns. A pre-trained sentence transformer encodes each profile into embeddings $\\mathbf{r}_{\\mathrm{short}}$ and $\\mathbf{r}_{\\mathrm{long}}$, and a single learnable attention vector computes weights $\\alpha_{\\mathrm{short}}$ and $\\alpha_{\\mathrm{long}}$ that combine the two into the final user embedding $\\mathbf{e}_u = \\alpha_{\\mathrm{short}} \\mathbf{r}_{\\mathrm{short}} + \\alpha_{\\mathrm{long}} \\mathbf{r}_{\\mathrm{long}}$. An MLP then scores the concatenation of $\\mathbf{e}_u$ with the item embedding. The attention weight is the interpretable part: it reports, per user, how much the recommendation relies on recent versus long-standing interests.","core_discovery":"The central claim is that separating a user's recent and enduring preferences into two LLM-written textual summaries, then merging them with an attention weight, yields a user representation that ranks held-out future interactions better than the common practice of averaging all interacted item embeddings. The authors report 17% higher Recall@10 and 14% higher NDCG@10 over the centric baseline on Movies&TV, where histories are large and tastes shift often, and smaller but still positive gains on Video Games, where histories are short and stable. The same outputs that drive the ranking—the two natural-language profiles and the attention split between them—double as the explanation, so transparency is a property of the model itself rather than a post-processing step.","pith_inferences":["A consequence the authors do not spell out: if the summaries are generated from training data only, the Movies&TV gains suggest the LLM summaries extract transferable signals; this could be tested by feeding longer histories and measuring when summarization cost dominates.","A natural user study: compare the learned attention weights with users' own stated split of recent versus lasting influence; agreement would strengthen the explainability claim beyond the paper's illustrative example.","The two-prompt summarization pattern should transfer to other content features such as reviews, tags, or genres; the architecture changes only in the input text."],"forward_implications":["In domains with large, frequently shifting user histories, the method's gains over averaged-embedding baselines are substantial: 17% Recall@10 and 14% NDCG@10 on Movies&TV.","Because the textual profiles and attention weights are readable by humans, the same pipeline can serve as an explanation interface with no additional explanation model.","In sparse, stable domains like Video Games, temporal text profiling retains an edge on Recall@20 but not consistently on other metrics, so the benefit depends on profile size and variability.","The ablation results indicate that both the temporal split (short vs. long term) and the nonlinear MLP scoring are needed for the full gains; removing either lowers accuracy."],"supporting_citations":[{"why":"Supplies the Movies&TV and Video Games datasets used for evaluation.","marker":"[8]"},{"why":"The GPT-4o-mini model that generates the short-term and long-term textual profiles.","marker":"[1]"},{"why":"The pre-trained BERT model used to encode profiles and item descriptions.","marker":"[5]"},{"why":"The specific SBERT variant (all-MiniLM-L6-v2) that produces the 384-dimensional embeddings.","marker":"[16]"},{"why":"The attention mechanism used to fuse short-term and long-term embeddings.","marker":"[22]"},{"why":"Documents the common centric practice of averaging item embeddings, which is the main baseline.","marker":"[6]"},{"why":"Motivates explainability as a requirement, framing the paper's contribution.","marker":"[23]"}],"fun_headline_variants":["LLM profiles boost recommendation recall by 17%, add explainability","Attention-fused LLM profiles give 17% recall gain, explainable","Short and long-term LLM summaries improve recall by 17%","Temporal LLM profiles: higher recall, transparent recommendations","LLM-written profiles explain and improve recommendations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume the LLM summaries are generated from each user's training interactions only; the paper never says that validation and test interactions are excluded, so the reported gains could partly reflect leakage of future items.","fun_headline_variants_meta":{"raw":{"variants":["LLM profiles boost recommendation recall by 17%, add explainability","Attention-fused LLM profiles give 17% recall gain, explainable","Short and long-term LLM summaries improve recall by 17%","Temporal LLM profiles: higher recall, transparent recommendations","LLM-written profiles explain and improve recommendations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000824,"raw_usage":{"total_tokens":3565,"prompt_tokens":865,"completion_tokens":2700,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":2614}},"tokens_in":481,"tokens_out":2700,"duration_ms":20244,"temperature":1.0,"reasoning_tokens":2614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:32:32.426149+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regenerate the short-term and long-term summaries using only the earliest 60% of each user's chronological history (the training split) and re-run the same ranking evaluation; if the improvement over the centric baseline drops or disappears, the original results were inflated by seeing held-out items.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the common centric practice of averaging item embeddings, which is the main baseline."}],"review_version":1}