Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM agents evolve recommenders using critiques, not just scores

desk verdict Real combination of user-simulator critique and co-evolving diagnostics inside LLM code evolution; standard-metric gains are credible, but the user-satisfaction claim is in-simulation, not independently validated. read the letter →

arxiv 2602.12612 v2 pith:CBFLNGWF submitted 2026-02-13 cs.IR cs.AI

Sein Kim , Sangwu Park , Hongseok Kang , Wonjoong Kim , Jimin Seo , Yeonjun In , Kanghoon Yoon , Hyunsik Jeon
show 1 more author
Chanyoung Park
This is my paper · ORCID
classification cs.IRcs.AI
keywords recommendersystemsLLM-drivencodeevolutiondirectionalfeedbackusersimulationmodeldiagnosisco-evolutionagenticAIself-evolvingagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that LLM-driven code evolution for recommender systems fails when guided only by scalar accuracy metrics, because NDCG collapses many distinct failure modes—popularity bias, category mismatch, recency blindness—into a single number. Self-EvolveRec adds two feedback channels: a User Simulator that critiques recommendation lists in plain language from persona-conditioned viewpoints, and a Model Diagnosis Tool that probes the model's internals for quantitative confirmation. The two channels are cross-checked, and the diagnosis tool itself evolves alongside the model so its probes match new architecture components. Experiments across four datasets and four seed recommenders report consistent gains over NAS and scalar-driven LLM evolution baselines, plus higher simulated user satisfaction.

What carries the argument

The directional feedback loop: a User Simulator (SIM) generates persona-conditioned natural-language critiques, while a Model Diagnosis Tool (DIAG) computes measurable internal probes such as embedding collapse (mean pairwise cosine similarity of item embeddings) and ranking margin (logit gap between ground-truth and negative items). An LLM planner turns both feedback streams into a development report, and a coding agent applies targeted code edits. The co-evolution step re-derives DIAG after each architecture change, ensuring the probes can verify newly added components and translate the simulator's qualitative complaints into concrete numerical signals.

What would settle it

Run Self-EvolveRec with an inverted user simulator whose critiques deliberately reward the opposite of genuine preference, and check whether standard NDCG and HR still rise; or deploy two evolved pipelines, one guided by simulator critiques and one by scalar metrics, in an online experiment comparing real-user retention and satisfaction. If simulated satisfaction diverges from real-user satisfaction, the directional loop's core claim fails.

Watch

Extended reading notes

Core claim

The central claim is that pairing qualitative user critiques with quantitative internal probes lets an LLM coding agent pinpoint root causes of failure and make targeted code changes, producing deterministic improvements in the recommendation pipeline rather than undirected trial-and-error. The Diagnosis Tool-Model Co-Evolution mechanism keeps verification criteria aligned with a shifting codebase: when the simulator reports a new failure mode such as order-insensitivity, the diagnostic tool autonomously generates corresponding probes, like swapping the last two items in a sequence and measuring ranking shifts. This cross-verification loop is what distinguishes the framework from scalar-metr

Load-bearing premise

The load-bearing premise is that LLM-generated critiques from persona-conditioned simulated users reflect real user preferences closely enough that optimizing for them improves real satisfaction and accuracy; the paper validates satisfaction with the same class of simulators, so this fidelity is assumed rather than demonstrated.

Editorial extensions

If this is right

  • Recommender optimization can escape fixed NAS operator pools: the search target becomes whole codebases, including losses, sampling logic, and encoders.
  • Evolution becomes cause-directed: each code change is traceable to a diagnosed failure mode, reducing wasteful exploration iterations.
  • Performance gains extend to user-centric metrics such as view ratio, satisfaction score, and browsing depth, not just ranking accuracy.
  • The verification mechanism stays valid as the architecture drifts, because the diagnostic tool co-evolves with the model.
  • The same pipeline works from a random recommender up to a strong ensemble, suggesting applicability across the service lifecycle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The critical open question is transfer to real users: satisfaction evidence uses simulated agents of the same kind that generate the training feedback, so a real-user A/B test would be the decisive next check.
  • Because the diagnostic tool makes feedback objectives explicit, its probes could double as auditable constraint signals to monitor or restrict evolved code.
  • Per-iteration cost is substantial; a promising efficiency test is predicting metric outcomes directly from proposed code diffs instead of retraining and re-evaluating every variant.
  • If the simulator's critique distribution is miscalibrated, the loop could overfit to simulated personas; injecting a known critique bias and measuring metric drift would reveal sensitivity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Self-EvolveRec, an LLM-driven evolutionary framework for recommender systems. It augments scalar-metric feedback (NDCG/HR) with two additional feedback channels: a User Simulator (SIM) that generates qualitative natural-language critiques from persona-conditioned LLM agents, and a Model Diagnosis Tool (DIAG) that computes quantitative structural probes such as embedding collapse and ranking margin. A 'Diagnosis Tool–Model Co-Evolution' step rewrites DIAG as the model evolves, introducing new metrics that encode SIM's qualitative complaints. The framework is evaluated on four datasets (Amazon CDs/Electronics/Office, MovieLens) with four seed recommenders (NCF, NGCF, SASRec, MoRec) against NAS and LLM-based code-evolution baselines. The paper reports consistent NDCG@5/HR@5 improvements over baselines, higher simulated user satisfaction, better LLM-as-a-judge code-quality scores, and case studies showing structured evolutionary trajectories.

Significance. If the claimed improvements are reproducible, Self-EvolveRec is a meaningful step toward open-ended recommender-pipeline optimization: the directional feedback loop is a sensible response to the limited diagnosability of scalar metrics, and the co-evolution of diagnostic tools is a novel mechanism. The empirical scope is a strength: four datasets, four seed architectures, ablations of each component, extreme-initialization tests, and injected-deficiency case studies. The code release and prompt templates are also valuable. However, the current evidence does not fully support the two headline claims: the word 'significantly' is not backed by statistical tests, and the user-satisfaction evaluation is partly circular because the same LLM-simulator paradigm provides both the optimization feedback and the evaluation.

major comments (4)
  1. [Abstract; §5.1.1, Table 1] The headline claim that Self-EvolveRec 'significantly outperforms' baselines is not supported by any significance test. Tables 1–6 report single point estimates with no variance, confidence intervals, or number of runs. Because the pipeline is stochastic at multiple levels (LLM sampling, neural-network training), differences of ~0.001–0.02 may be noise. Please report means±std over at least 3 seeds and paired significance tests against each baseline (or equivalently soften the claim to 'consistently improves in these runs'). This is load-bearing for the abstract.
  2. [§5.1.2; §4.1.1; §4.3] The user-satisfaction claim is circular. The in-loop SIM uses the same agentic LLM user-simulation paradigm (Agent4Rec/PUB) as the evaluators in Table 2, so optimizing against SIM teaches the model to satisfy that simulator class; Table 2 then measures alignment with a similar simulator, not real users. Moreover, DIAG co-evolution (Sec 4.3) constructs new diagnostic metrics from RSIM's qualitative critiques, so DIAG does not independently verify SIM—it transduces SIM's opinions into numeric form. Please present Table 2 as 'simulated user satisfaction' and either add external validation (real-user study, logged behavior, or a different simulation paradigm) or carefully hedge the abstract's satisfaction claim.
  3. [§5.3.5; App. F.1.2; Table 4] The comparison is confounded by unequal compute. Self-EvolveRec uses ~25 min/iteration vs ~6 min (AlphaEvolve) and ~14 min (DeepEvolve) (Table 8). The paper argues fewer iterations to peak, but does not report total compute or equalize the LLM-inference/training budget. The observed gains may partly reflect greater computational cost rather than directional feedback. Please add an equal-compute comparison (same wall-clock or same number of LLM calls) or an analysis isolating the effect of the directional signal.
  4. [§5.1.3; App. G.2] The codebase-quality evaluation uses GPT-5 as LLM-as-a-judge—the same model family that generated the evolved code—without human validation or inter-rater agreement. The judge's criteria overlap with the instructions used in the evolution loop, creating a self-evaluation bias. The claim that Self-EvolveRec produces higher 'Creativity'/'Insight'/'Personalization' would be stronger with human expert ratings on a sample of evolved codebases, or at least an independent judge model and reported agreement.
minor comments (5)
  1. [§5 Datasets] Typo: 'Amzaon' should be 'Amazon'.
  2. [Figure 5 caption] The caption contains stray Korean text ('진짜진짜진짜진짜 최종') that should be removed.
  3. [Table 2] Satisfaction results are reported only for SASRec and NCF seeds; please justify the omission of NGCF and MoRec, or report them.
  4. [§5.3.5] The phrase 'deterministic improvements' overstates the determinism of an LLM-based pipeline; consider 'consistent' or 'robust' improvements.
  5. [Eq. (1)] Equation (1) presents a bi-level optimization problem that the evolutionary search does not actually solve; the text should clarify that this is a goal specification, not the optimization procedure.

Circularity Check

2 steps flagged · score 6.0 of 10

User-satisfaction evaluation and DIAG verification are partly circular; NDCG/HR gains remain independent.

  1. self definitional [Sec. 4.3 (Diagnosis Tool - Model Co-Evolution, Eq. 5-6); Sec. 4.1.2]
    "Second, DIAG serves to quantitatively verify the qualitative insights provided by the SIM. ... Therefore, DIAG evolves to stay compatible with the evolved codebase while formulating specific metrics that mathematically capture the essence of the SIM's feedback."

    DIAG(t+1) is constructed (Eq. 6) using R_SIM, R_Analyze, and K_DIAG, so the diagnostic metrics are defined from the very SIM critiques they are later used to verify. Sec. 4.1.2 states that DIAG 'serves as a verification mechanism to check whether the qualitative deficiencies pointed out by SIM have actually resolved.' A verifier built from the feedback it verifies cannot independently confirm that feedback; agreement is encoded by construction. This does not make the Table 1 NDCG/HR benchmark circular, but it undermines the paper's claim that DIAG provides independent 'quantitative verification' of the SIM's qualitative critiques.

  2. other [Sec. 5.1.2 (User Satisfaction Analysis) vs Sec. 4.1.1 (User Simulator)]
    "To bridge the gap between static metrics and actual user satisfaction, we adopt the agentic simulation environment from Agent4Rec [50] and PUB [31] as a scalable proxy for A/B testing. ... By adopting the agentic paradigm [31, 50], ... our simulator acts as a diverse set of virtual users..."

    The in-loop User Simulator is built from the same agentic LLM-simulator paradigm, and the same cited systems (Agent4Rec/PUB), that are used for the user-satisfaction evaluation. Since code evolution is explicitly conditioned on the SIM's critiques (R_SIM), the Table 2 satisfaction metrics measure how well the evolved model satisfies the same proxy family it was optimized against. This is essentially in-distribution evaluation for the satisfaction claim, not an independent human-validated measure. The NDCG@5/HR@5 results in Table 1 are separate and remain independent evidence.

full rationale

The headline claim has two components. The standard-metric component is not circular: the optimization objective in Eq. 1 is a standard ranking metric, and Table 1 compares against external NAS and LLM-evolution baselines with no fitted parameters from the evaluation. That part stands. Two other parts are circular in different degrees. First, the Model Diagnosis Tool is presented as a deterministic, independent verifier of the User Simulator's qualitative feedback, but Sec. 4.3 explicitly says the co-evolved DIAG is created by 'formulating specific metrics that mathematically capture the essence of the SIM's feedback,' and Eq. 6 feeds R_SIM into DIAG creation. The verifier is therefore a transduction of the critique it verifies; any 'confirmation' is built into the metric design. This affects the paper's directional-feedback and 'deterministic improvement' narrative, though not the NDCG/HR benchmark. Second, the user-satisfaction evaluation uses the same agentic simulation paradigm (Agent4Rec/PUB) from which the in-loop SIM is adopted, and the code evolution is explicitly driven by the SIM's critiques. The Table 2 satisfaction results are thus an in-family proxy evaluation rather than an independent check against real users. This weakens the 'user satisfaction' half of the central claim but does not make the standard-metric comparisons circular. No load-bearing self-citation or uniqueness theorem import is present, so the score reflects partial circularity, not full equivalence.

Assumptions & free parameters 4 free parameters · 6 assumptions · 3 invented entities

The central claim rests on several domain assumptions that are plausible but unverified: LLM user simulators behave like real users, LLM coding agents make correct edits, and LLM-generated diagnostic probes are valid indicators of model health. The method introduces three soft entities (SIM, DIAG, co-evolution) whose independent evidence is mostly internal to the paper. The LLM backend, user-sample size, trait thresholds, and stopping criterion are hand-chosen factors that the measured performance depends on.

free parameters (4)
  • Number of sampled users |U_sample| = 20
    Set in App. C; sensitivity analysis in Fig. 6 shows performance stabilizes around 5, so 20 is a safety margin. The summarization of qualitative feedback (Eq. 2) depends on this sample size.
  • Trait quantile thresholds (LOW/MID/HIGH for Activity, Conformity, Diversity) = quantile thresholds not numerically reported
    App. B defines the mapping from trait scores to discrete levels; the exact quantile cutoffs are hand-chosen and not ablated.
  • Maximum evolution iterations = 21
    App. C caps all LLM-evolution runs at 21; peak-performance and FAIL outcomes (Table 4) are stopping-point-dependent.
  • LLM backend selection = GPT-5-mini (planner/simulator), GPT-5 (coder/judge)
    App. C chooses these models; all results could shift with weaker or stronger LLMs, so this is a hand-chosen dependence of the central claim.
assumptions (6)
  • domain assumption Simulated-user critiques are faithful proxies for real user preferences and failure modes.
    Sec. 4.1.1 builds personas from Activity/Conformity/Diversity and interaction history; Sec. 5.1.2 evaluates satisfaction with the same class of LLM simulators, not real users.
  • domain assumption GPT-5 can make correct, executable SEARCH/REPLACE edits without breaking the codebase.
    The whole evolution pipeline (B(t+1)=LLM(ICODE, ...)) assumes the coding agent's edits are semantically faithful and syntactically valid; no reliability statistics are provided for edit success.
  • domain assumption Hand-designed and co-evolved diagnostic probes accurately indicate underlying model deficiencies.
    Seed probes (embedding collapse, ranking margin) and co-evolved probes (swap sensitivity, off-category rate) are validated only through author-injected deficiency case studies (Fig. 4, Fig. 7), not by an independent ground truth.
  • domain assumption Retrieved academic knowledge from online RAG is relevant and correctly integrated by the planner.
    The planning step (Eq. 4) relies on K retrieved from online sources; no audit of retrieval quality or alignment with the diagnosed failure is provided.
  • domain assumption NDCG@5 and HR@5 with 99 random negatives are valid proxies for recommendation quality.
    This is the standard leave-last-out protocol from prior work (Sec. 5), and it is the ultimate objective in Eq. 1; it may not capture all user-centric quality dimensions.
  • domain assumption LLM-as-a-Judge scores correlate with actual code quality and user satisfaction.
    Code quality in Sec. 5.1.3 is evaluated by GPT-5 with no inter-judge agreement or human validation; satisfaction is evaluated by Agent4Rec/PUB simulators with no real-user calibration.
invented entities (3)
  • User Simulator (SIM)
    purpose: Generate qualitative natural-language critiques from simulated user personas to guide code evolution (Sec. 4.1.1).
    Reliability is assessed only by internal target-item identification (Table 6) and by agentic simulators of the same type; no external real-user validation is provided.
  • Model Diagnosis Tool (DIAG)
    purpose: Probe model internals (embedding collapse, ranking margin) to quantitatively verify failures (Sec. 4.1.2).
    Validity is shown via author-injected deficiency case studies (Fig. 4, Fig. 7), not by an independent ground-truth benchmark of diagnostic correctness.
  • Diagnosis Tool-Model Co-Evolution
    purpose: Regenerate diagnostic probes as the architecture changes and to quantify SIM critiques (Sec. 4.3).
    The co-evolved metrics are generated by the same LLM family and are not formally specified beyond prompts; there is no external validation that the new probes measure what they claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback." pith.science (2026). https://pith.science/paper/CBFLNGWF

@misc{pith2026260212612,
  author       = {Pith},
  title        = {Pith review of: Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CBFLNGWF}},
  note         = {Machine review of arXiv:2602.12612}
}
read the original abstract

Traditional methods for automating recommender system design, such as Neural Architecture Search (NAS), are often constrained by a fixed search space defined by human priors, limiting innovation to pre-defined operators. While recent LLM-driven code evolution frameworks shift fixed search space target to open-ended program spaces, they primarily rely on scalar metrics (e.g., NDCG, Hit Ratio) that fail to provide qualitative insights into model failures or directional guidance for improvement. To address this, we propose Self-EvolveRec, a novel framework that establishes a directional feedback loop by integrating a User Simulator for qualitative critiques and a Model Diagnosis Tool for quantitative internal verification. Furthermore, we introduce a Diagnosis Tool - Model Co-Evolution strategy to ensure that evaluation criteria dynamically adapt as the recommendation architecture evolves. Extensive experiments demonstrate that Self-EvolveRec significantly outperforms state-of-the-art NAS and LLM-driven code evolution baselines in both recommendation performance and user satisfaction. Our code is available at https://github.com/Sein-Kim/self_evolverec.

Figures

Figures reproduced from arXiv: 2602.12612 by the authors.

Figure 1
Figure 1. Two core mechanism of Self-EvolveRec. (a) is the overview of the Directional Feedback Generation: (a.1) is the user simulator, (a.2) is the model diagnosis tool. (b) is the Diagnosis Tool - Model Co-evolution in Self-EvolveRec. LLM-based User Simulation. LLM-based user simulation has emerged as a promising alternative to static recommendation met￾rics and costly online A/B testing. Initial approaches focused on real… view at source ↗
Figure 2
Figure 2. Overall evolutionary pipeline of Self-EvolveRec. 1) Embedding Collapse: To detect a state where representations degenerate into a narrow subspace losing discriminative power, DIAG computes the mean pairwise cosine similarity across sampled item embeddings. A high similarity score serves as a proxy for representation degeneration. 2) Ranking Margin: DIAG evaluates decision boundaries by an￾alyzing the ranking margin … view at source ↗
Figure 3
Figure 3. LLM-as-a-Judge evaluation of the evolved models. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Case study on Diagnosis Tool - Model Co-Evolution [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Case study on evolutionary trajectory on CDs [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Recommendation performance over number of [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Case study on Diagnosis Tool - Model Co-Evolution [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Case study on evolutionary trajectory on CDs [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Example prompt of ISIM [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Example prompt of ISUMMARIZE [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Example prompt of IDIAG [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Example prompt of IPLAN [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Example prompt of IREPORT [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Example prompt of ICODE [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Example prompt of IAnalyze [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Example prompt of IPLAN-DIAG [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Example prompt of IREPORT-DIAG [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Example prompt of ICODE-DIAG [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: Example prompt of ICODE-SIM [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Example prompt of LLM-as-a-Judge for evolved models evaluation. [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SR-Agent: An Experience-Driven Agentic Framework for Post-Ranking Strategy Refinement in E-Commerce Recommendation

    cs.AI 2026-07 conditional novelty 5.0 of 10

    SR-Agent is an LLM-agent pipeline that automatically inspects, diagnoses, and updates post-ranking recommendation strategies; a one-month Kuaishou A/B test reports +0.71% order volume, +0.34% browsing depth, and +0.48...

Reference graph

Works this paper leans on

84 extracted references · 10 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Mehdi Ben Ayed, Fei Feng, Jay Adams, Vishwakarma Singh, Kritarth Anand, and Jiajing Xu. 2025. RecoMind: A reinforcement learning framework for opti- mizing in-session user satisfaction in recommendation systems.arXiv preprint arXiv:2508.00201(2025)

    arXiv 2025

  2. [2]

    Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al

  3. [4]

    Yuanzheng Ci, Chen Lin, Ming Sun, Boyu Chen, Hongwen Zhang, and Wanli Ouyang. 2021. Evolving search space for neural architecture search. InProceedings of the IEEE/CVF International Conference on Computer Vision. 6659–6669

    2021

  4. [6]

    Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural architecture search: A survey.Journal of Machine Learning Research20, 55 (2019), 1–21

    2019

  5. [7]

    Francesco Fabbri, Gustavo Penha, Edoardo D’Amico, Alice Wang, Marco De Nadai, Jackie Doremus, Paul Gigioli, Andreas Damianou, Oskar Stål, and Mounia Lalmas

  6. [8]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997 2, 1 (2023)

    arXiv 2023

  7. [9]

    Goldberg

    Lewis R. Goldberg. 1992. THE DEVELOPMENT OF MARKERS FOR THE BIG- FIVE FACTOR STRUCTURE.Psychological Assessment4 (1992), 26–42. https: //api.semanticscholar.org/CorpusID:144709415

    1992

  8. [10]

    Gomez-Uribe and Neil Hunt

    Carlos A. Gomez-Uribe and Neil Hunt. 2016. The Netflix Recommender System: Algorithms, Business Value, and Innovation. 6, 4 (2016). doi:10.1145/2843948

    doi:10.1145/2843948 2016

Show all 84 references
  1. [11]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A survey on llm-as-a-judge.The Innovation(2024)

    2024

  2. [12]

    Maxwell Harper and Joseph A

    F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context.ACM Trans. Interact. Intell. Syst.5, 4, Article 19 (Dec. 2015), 19 pages. doi:10.1145/2827872

    2015 doi

  3. [13]

    Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. Lightgcn: Simplifying and powering graph convolution network for recommendation. InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 639–648

    2020

  4. [14]

    Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. InProceedings of the 26th international conference on world wide web. 173–182

    2017

  5. [15]

    Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley

  6. [16]

    Ganesh Jawahar, Muhammad Abdul-Mageed, Laks Lakshmanan, and Dujian Ding. 2024. Llm performance predictors are good initializers for architecture search. InFindings of the Association for Computational Linguistics: ACL 2024. 10540–10560

    2024

  7. [17]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In2018 IEEE international conference on data mining (ICDM). IEEE, 197–206

    2018

  8. [18]

    Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, and Hwanjo Yu

  9. [19]

    Jiwan Kim, Hongseok Kang, Sein Kim, Kibum Kim, and Chanyoung Park. 2025. Disentangling and Generating Modalities for Recommendation in Missing Modal- ity Scenarios. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieva...

    2025

  10. [20]

    Kibum Kim, Sein Kim, Hongseok Kang, Jiwan Kim, Heewoong Noh, Yeonjun In, Kanghoon Yoon, Jinoh Oh, and Chanyoung Park. 2025. Image is All You Need: Towards Efficient and Effective Large Language Model-Based Recommender Systems.arXiv preprint arXiv:2503.06238(2025)

    2025

  11. [21]

    Sein Kim, Hongseok Kang, Seungyoon Choi, Donghyun Kim, Minchul Yang, and Chanyoung Park. 2024. Large Language Models meet Collaborative Filtering: An Efficient All-round LLM-based Recommender System. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Da...

    2024

  12. [22]

    Association for Computing Machinery, New York, NY, USA

    Convolutional Matrix Factorization for Document Context-Aware Recom- mendation(RecSys ’16). Association for Computing Machinery, New York, NY, USA. doi:10.1145/2959100.2959165

  13. [23]

    Sein Kim, Namkyeong Lee, Donghyun Kim, Minchul Yang, and Chanyoung Park. 2023. Task Relation-aware Continual User Representation Learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining(Long Beach, CA, USA)(KDD ’23). Association for Com...

    2023

  14. [24]

    Wonjoong Kim, Sangwu Park, Yeonjun In, Sein Kim, Dongha Lee, and Chanyoung Park. 2025. Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents.arXiv preprint arXiv:2510.02837(2025)

    2025 arXiv

  15. [25]

    Ravi Krishna, Aravind Kalaiah, Bichen Wu, Maxim Naumov, Dheevatsa Mudigere, Misha Smelyanskiy, and Kurt Keutzer. 2021. Differentiable nas framework and application to ads ctr prediction.arXiv preprint arXiv:2110.14812(2021)

    2021 arXiv

  16. [26]

    Sein Kim, Hongseok Kang, Kibum Kim, Jiwan Kim, Donghyun Kim, Minchul Yang, Kwangjin Oh, Julian McAuley, and Chanyoung Park. 2025. Lost in Sequence: Do Large Language Models Understand Sequential Recommendation?. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Disc...

    2025

  17. [27]

    Zelong Li, Jianchao Ji, Yingqiang Ge, and Yongfeng Zhang. 2022. Autolossgen: Automatic loss function generation for recommender systems. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1304–1315

    2022

  18. [28]

    Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xi- uqiang He, Zhenguo Li, and Yong Yu. 2020. Autofis: Automatic feature interaction selection in factorization models for click-through rate prediction. Inproceedings of the 26th ACM SIGKDD international ...

    2020

  19. [29]

    Fei Liu, Xinyu Lin, Hanchao Yu, Mingyuan Wu, Jianyu Wang, Qiang Zhang, Zhuokai Zhao, Yinglong Xia, Yao Zhang, Weiwei Li, et al. 2025. Recoworld: Build- ing simulated environments for agentic recommender systems.arXiv preprint arXiv:2509.10397(2025)

    2025 arXiv

  20. [30]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing s...

    2020

  21. [31]

    Chenglong Ma, Ziqi Xu, Yongli Ren, Danula Hettiachchi, and Jeffrey Chan. 2025. PUB: an LLM-enhanced personality-driven user behaviour simulator for recom- mender system evaluation. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Info...

    2025

  22. [32]

    Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Eureka: Human-level reward design via coding large language models.arXiv preprint arXiv:2310.12931(2023)

    2023 arXiv

  23. [33]

    McNee, John Riedl, and Joseph A

    Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being accurate is not enough: how accuracy metrics have hurt recommender systems. InCHI ’06 Extended Abstracts on Human Factors in Computing Systems(Montréal, Québec, Canada)(CHI EA ’06). Association for Computing Machine...

    2006

  24. [34]

    Gang Liu, Yihan Zhu, Jie Chen, and Meng Jiang. 2025. Scientific algorithm discovery by augmenting alphaevolve with deep research.arXiv preprint arXiv:2510.06056(2025)

    2025

  25. [35]

    Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole- Jean Wu, Alisson G Azzolini, et al. 2019. Deep learning recommendation model for personalization and recommendation systems.arXiv preprin...

    2019 arXiv

  26. [36]

    Alexander Novikov, Ngân V ˜u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. 2025. AlphaEvolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv...

    2025 arXiv

  27. [37]

    Esteban Real, Chen Liang, David So, and Quoc Le. 2020. Automl-zero: Evolving machine learning algorithms from scratch. InInternational conference on machine learning. PMLR, 8007–8019

    2020

  28. [38]

    Manel Mezghani, Corinne Amel Zayani, Ikram Amous, and Faiez Gargouri

  29. [39]

    Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024. Mathematical discoveries from program search with large language models.Nature625, ...

    2024

  30. [40]

    Ruslan Salakhutdinov and Andriy Mnih. 2007. Probabilistic Matrix Factorization (NIPS’07). Curran Associates Inc., Red Hook, NY, USA. Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

    2007

  31. [41]

    2025.OpenEvolve: an open-source evolutionary coding agent

    Asankhaya Sharma. 2025.OpenEvolve: an open-source evolutionary coding agent. https://github.com/algorithmicsuperintelligence/openevolve

    2025

  32. [42]

    Qingquan Song, Dehua Cheng, Hanning Zhou, Jiyan Yang, Yuandong Tian, and Xia Hu. 2020. Towards automated neural interaction discovery for click-through rate prediction. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 945–955

    2020

  33. [43]

    Schwartz, and Ariel Knafo

    Sonia Roccas, Lilach Sagiv, Shalom H. Schwartz, and Ariel Knafo. 2002. The Big Five Personality Factors and Personal Values.Personality and Social Psychology Bulletin28 (2002), 789 – 801. https://api.semanticscholar.org/CorpusID:144611052

    2002

  34. [44]

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang

  35. [45]

    Holland, and Quentin F

    Reiko Tanese, John H. Holland, and Quentin F. Stout. 1989.Distributed genetic algorithms for function optimization. Ph. D. Dissertation. USA. AAI9001722

    1989

  36. [46]

    Jiaxi Tang and Ke Wang. 2018. Personalized top-n sequential recommenda- tion via convolutional sequence embedding. InProceedings of the eleventh ACM international conference on web search and data mining. 565–573

    2018

  37. [47]

    Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. InProceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval. 165–174

    2019

  38. [48]

    Harald Steck. 2018. Calibrated recommendations. InProceedings of the 12th ACM Conference on Recommender Systems(Vancouver, British Columbia, Canada) (RecSys ’18). Association for Computing Machinery, New York, NY, USA, 154–162. doi:10.1145/3240323.3240372

    2018

  39. [49]

    Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. 2023. Where to Go Next for Recommender Systems? ID- vs. Modality-based Recommender Models Revisited. InProceedings of the 46th International ACM SIGIR Conference on Research and Devel...

    2023

  40. [50]

    An Zhang, Yuxin Chen, Leheng Sheng, Xiang Wang, and Tat-Seng Chua. 2024. On generative agents in recommendation. InProceedings of the 47th international ACM SIGIR conference on research and development in Information Retrieval. 1807– 1817

    2024

  41. [51]

    Tunhou Zhang, Dehua Cheng, Yuchen He, Zhengxing Chen, Xiaoliang Dai, Liang Xiong, Feng Yan, Hai Li, Yiran Chen, and Wei Wen. 2023. NASRec: weight sharing neural architecture search for recommender systems. InProceedings of the ACM Web Conference 2023. 1199–1207

    2023

  42. [52]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

    2023

  43. [53]

    Ruiqi Zheng, Liang Qu, Bin Cui, Yuhui Shi, and Hongzhi Yin. 2023. Automl for deep recommender systems: A survey.ACM Transactions on Information Systems 41, 4 (2023), 1–38

    2023

  44. [54]

    Xuesi Wang, Guangda Huzhang, Qianying Lin, and Qing Da. 2022. Learning-to- ensemble by contextual rank aggregation in e-commerce. InProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 1036– 1044

    2022

  45. [55]

    Barret Zoph and Quoc Le. 2017. Neural Architecture Search with Reinforce- ment Learning. InInternational Conference on Learning Representations. https: //openreview.net/forum?id=r1Ue8Hcxg

    2017

  46. [56]

    lack of diversity

    Kuan Zou and Aixin Sun. 2025. A Survey of Real-World Recommender Sys- tems: Challenges, Constraints, and Industrial Perspectives.arXiv preprint arXiv:2509.06002(2025). Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Kim et al. A Ethics Statement To the best of our know...

    2025 arXiv

  47. [60]

    McNee, Joseph A

    Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. 2005. Improving recommendation lists through topic diversification. InProceedings of the 14th International Conference on World Wide Web(Chiba, Japan)(WWW ’05). Association for Computing Machinery, New Yo...

    2005

  48. [66]

    Generate and evaluate ideas -For each idea, critically assess as a senior researcher with one positive and one negative reason: -Originality (0-10): Is the idea new? Is the idea a novel combination of well-known techniques? Is it clearly different from previous contributions? ...

  49. [67]

    Being concise with 200-500 words without trivial and redundant information

    Write the report in Markdown For the selected idea, include: -A synthesis of insights and proposed directions -The structured framework of existing methods and the new algorithm -A list of new ideas with their assessment scores -Detailed description of the chosen/best idea, in...

    2018

  50. [78]

    good” vs “bad

    Preserve the original code's indentation and formatting. Place the lines of `### >>> Self-EvolveRec-BLOCK-START: <research idea>` and `### <<< Self-EvolveRec-BLOCK-END` at the same indentation level as the code they annotate. Figure 14: Example prompt ofI CODE. Self-EvolveRec:...

    2018

  51. [79]

    For each insight, explain in 2-3 sentences how it relates to the target problem

    Extract insights Identify 3-5 scientific insights from the starting point and 3-5 from related works. For each insight, explain in 2-3 sentences how it relates to the target problem

  52. [80]

    Organize research directions Group the insights into 3-5 coherent directions (for example, learning objectives, model classes, or optimization methods)

  53. [81]

    Build a structured framework Create a conceptual map (such as a taxonomy, grid, or matrix) that unifies existing methods, reveals patterns, and highlights gaps

  54. [82]

    Generate and evaluate ideas -For each idea, critically assess as a senior researcher with one positive and one negative reason: -Originality (0-10): Is the idea new? Is the idea a novel combination of well-known techniques? Is it clearly different from previous contributions? ...

  55. [83]

    Being concise with 200-500 words without trivial and redundant information

    Write the report in Markdown For the selected idea, include: -A synthesis of insights and proposed directions -The structured framework of existing methods and the new algorithm -A list of new ideas with their assessment scores -Detailed description of the chosen/best idea, in...

    2018

  56. [94]

    Place the lines of `### >>> Self-EvolveRec-BLOCK-START: <research idea>` and `### <<< Self-EvolveRec-BLOCK-END` at the same indentation level as the code they annotate

    Preserve the original code's indentation and formatting. Place the lines of `### >>> Self-EvolveRec-BLOCK-START: <research idea>` and `### <<< Self-EvolveRec-BLOCK-END` at the same indentation level as the code they annotate. Figure 18: Example prompt ofI CODE-DIAG. Self-Evolv...

    2018

  57. [95]

    Think before coding, understand the research idea and current performance bottlenecks

  58. [96]

    Propose specific, actionable changes that are aligned with the target metrics

  59. [97]

    You may suggest multiple improvements beyond the research idea based on your understanding of optimization and machine learning

  60. [98]

    -If a NEW parameter has a default value of None, confirm that passing a non-None value triggers the intended code path

    When you are updating the code, please check the following: -When a NEW parameter or behavior is added, verify it is invoked in all call sites or in the overall workflow. -If a NEW parameter has a default value of None, confirm that passing a non-None value triggers the intend...

  61. [99]

    All `SEARCH` blocks must match the original code exactly

  62. [100]

    When you need to modify code that is not already inside a `Self-EvolveRec` block, wrap your changes with `### >>> Self-EvolveRec-BLOCK-START: <research idea>` and `### <<< Self-EvolveRec-BLOCK-END` markers

  63. [101]

    If you are updating code that is already marked by a `Self-EvolveRec` block, edit only the lines within that block and adjust the existing modification comment to reflect your new change

  64. [102]

    Each region you modify should have exactly one pair of start/end markers

    Do NOT nest one `Self-EvolveRec` block inside another. Each region you modify should have exactly one pair of start/end markers. …

  65. [103]

    Do not rewrite the entire file

    Limit your changes to what is strictly necessary. Do not rewrite the entire file

  66. [104]

    Ensure that all modified code remains correct and consistent, including any function signatures, parameter lists, and calls

  67. [105]

    Place the lines of `### >>> Self-EvolveRec-BLOCK-START: <research idea>` and `### <<< Self-EvolveRec-BLOCK-END` at the same indentation level as the code they annotate

    Preserve the original code's indentation and formatting. Place the lines of `### >>> Self-EvolveRec-BLOCK-START: <research idea>` and `### <<< Self-EvolveRec-BLOCK-END` at the same indentation level as the code they annotate. Figure 19: Example prompt ofI CODE-SIM. Conference ...

    2018

  68. [106]

    High (8-10): Designing completely new loss functions, implementing dynamic sampling strategies, or creating novel attention/interaction mechanisms

    Algorithmic Novelty -Definition: Does the code introduce genuinely new mechanisms rather than just tuning existing ones? -Checkpoints: Low (1-3): Simple hyperparameter tuning, variable renaming, or mere code refactoring without functional changes. High (8-10): Designing comple...

  69. [107]

    Black Box

    Logic Explicitness (Interpretability) -Definition: By reading the `forward` function, how easy is it to understand the decision mechanism? -Checkpoints: Low (1-3): "Black Box" logic. Data flows through deep, opaque layers making it hard to trace why a specific item was recomme...

  70. [108]

    Adding standard layers or changing parameters without a clear purpose or connection to a specific problem

    Problem-Solving Insight -Definition: Does the modification reflect a clear, logical intention to solve a specific recommendation problem (e.g., sparsity, bias, diversity)? -Checkpoints: Low (1-3): Random or generic changes. Adding standard layers or changing parameters without...

  71. [109]

    The model relies solely on `User_Embedding(dot product) Item_Embedding`

    User-Centric Adaptation (Personalization Depth) -Definition: Does the code treat the user merely as a static index (ID), or does it actively adapt to the user's specific context and constraints (e.g., recent history, price sensitivity, category affinity)? -Checkpoints: Low (1-...

  72. [2012]

    Association for Computing Machinery, New York, NY, USA

    A user profile modelling using social annotations: a survey(WWW ’12 Companion). Association for Computing Machinery, New York, NY, USA. doi:10.1145/2187980.2188230

  73. [2016]

    InProceedings of the 1st workshop on deep learning for recommender systems

    Wide & deep learning for recommender systems. InProceedings of the 1st workshop on deep learning for recommender systems. 7–10

  74. [2019]

    InProceedings of the 28th ACM international conference on information and knowledge management

    BERT4Rec: Sequential recommendation with bidirectional encoder rep- resentations from transformer. InProceedings of the 28th ACM international conference on information and knowledge management. 1441–1450

  75. [2024]

    Bridging Language and Items for Retrieval and Recommendation.arXiv preprint arXiv:2403.03952(2024)

    2024 arXiv

  76. [2025]

    InProceedings of the Nineteenth ACM Conference on Recommender Systems

    Evaluating podcast recommendations with profile-aware llm-as-a-judge. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 1181– 1186

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.