{"id":"59e749be-2a9a-4f4e-9dbb-5b5e95dfb033","arxiv_id":"2602.12612","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-driven recommender evolution loop that combines simulated-user critiques with co-evolving diagnostic probes outperforms scalar-metric-only evolution baselines on standard ranking metrics.","lead":"Self-EvolveRec is a system that uses an LLM to rewrite recommender code over many rounds, guided by simulated users' written complaints and by internal model-health checks. It matters because it replaces human trial-and-error tuning of recommendation pipelines with automated, explainable feedback.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"User-satisfaction results are circular: the same LLM-simulator paradigm provides both the optimization feedback and the evaluation, so the headline satisfaction claim is not independently validated.","rationale":"I read the paper as making two connected claims: (1) LLM-driven code evolution with qualitative user-simulator feedback improves standard recommendation metrics, and (2) it also improves real user satisfaction. The standard-metric results are consistent and the ablations support the contribution of each component, so I do not see a load-bearing error in the recommendation-performance claim. The weakest point is the satisfaction claim, because the optimization signal and the evaluation signal come from the same LLM-simulator paradigm. The reader's weakest assumption identified exactly this proxy-fidelity issue, and the DIAG co-evolution mechanism (Sec. 4.3) makes it worse by converting SIM critiques into diagnostic metrics rather than providing an independent check. I considered other concerns—absence of error bars, use of OpenEvolve as a stand-in for AlphaEvolve, and the non-monotonic trajectory in Fig. 5—but the simulator circularity is the most load-bearing because it directly supports the user-satisfaction half of the abstract's claim. A clean out-of-family or human evaluation would settle whether the loop is optimizing real preferences or its own simulation.","tokens_in":30422,"tokens_out":8519,"duration_ms":82080,"concrete_test":"Re-evaluate the final models from Table 1 (Self-EvolveRec and baselines) on the three satisfaction metrics using an out-of-family user simulator that was not used anywhere in the evolution loop—e.g., a separately built LLM simulator with a different model family, trait schema, and prompt style, or a small human panel rating the same top-5 lists on CDs/Electronics. If Self-EvolveRec's satisfaction gains over AlphaEvolve/DeepEvolve persist, the concern is resolved; if they shrink or reverse, the reported satisfaction results are an artifact of the shared simulator proxy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; Sec. 5.1.2) is that Self-EvolveRec improves both recommendation performance and user satisfaction. The user-satisfaction evidence, however, is generated by the same class of LLM-based generative agents (Agent4Rec/PUB) that the method is built on: the in-loop SIM (Sec. 4.1.1) uses the same agentic LLM-simulator paradigm as the evaluators in Sec. 5.1.2. Moreover, DIAG co-evolution (Sec. 4.3) builds new diagnostic metrics directly from the RSIM critiques, so DIAG does not independently verify the SIM's qualitative feedback—it transduces that feedback into numeric probes. The loop can therefore converge to a codebase that satisfies its own simulator, with no real-user validation to show that those simulated preferences are faithful to human users. If the simulator systematically over-penalizes diversity, hallucinates failure modes, or otherwise diverges from human judgment, the 'directional feedback' optimizes a biased objective and the user-satisfaction portion of the headline claim is unsupported. This does not invalidate the standard-metric gains in Table 1, which are the strongest independent evidence; it undercuts the 'user satisfaction' half of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Self-EvolveRec, an LLM-driven evolutionary framework for recommender systems. It augments scalar-metric feedback (NDCG/HR) with two additional feedback channels: a User Simulator (SIM) that generates qualitative natural-language critiques from persona-conditioned LLM agents, and a Model Diagnosis Tool (DIAG) that computes quantitative structural probes such as embedding collapse and ranking margin. A 'Diagnosis Tool–Model Co-Evolution' step rewrites DIAG as the model evolves, introducing new metrics that encode SIM's qualitative complaints. The framework is evaluated on four datasets (Amazon CDs/Electronics/Office, MovieLens) with four seed recommenders (NCF, NGCF, SASRec, MoRec) against NAS and LLM-based code-evolution baselines. The paper reports consistent NDCG@5/HR@5 improvements over baselines, higher simulated user satisfaction, better LLM-as-a-judge code-quality scores, and case studies showing structured evolutionary trajectories.","tokens_in":30710,"tokens_out":7491,"duration_ms":61283,"significance":"If the claimed improvements are reproducible, Self-EvolveRec is a meaningful step toward open-ended recommender-pipeline optimization: the directional feedback loop is a sensible response to the limited diagnosability of scalar metrics, and the co-evolution of diagnostic tools is a novel mechanism. The empirical scope is a strength: four datasets, four seed architectures, ablations of each component, extreme-initialization tests, and injected-deficiency case studies. The code release and prompt templates are also valuable. However, the current evidence does not fully support the two headline claims: the word 'significantly' is not backed by statistical tests, and the user-satisfaction evaluation is partly circular because the same LLM-simulator paradigm provides both the optimization feedback and the evaluation.","major_comments":[{"comment":"The headline claim that Self-EvolveRec 'significantly outperforms' baselines is not supported by any significance test. Tables 1–6 report single point estimates with no variance, confidence intervals, or number of runs. Because the pipeline is stochastic at multiple levels (LLM sampling, neural-network training), differences of ~0.001–0.02 may be noise. Please report means±std over at least 3 seeds and paired significance tests against each baseline (or equivalently soften the claim to 'consistently improves in these runs'). This is load-bearing for the abstract.","section":"Abstract; §5.1.1, Table 1"},{"comment":"The user-satisfaction claim is circular. The in-loop SIM uses the same agentic LLM user-simulation paradigm (Agent4Rec/PUB) as the evaluators in Table 2, so optimizing against SIM teaches the model to satisfy that simulator class; Table 2 then measures alignment with a similar simulator, not real users. Moreover, DIAG co-evolution (Sec 4.3) constructs new diagnostic metrics from RSIM's qualitative critiques, so DIAG does not independently verify SIM—it transduces SIM's opinions into numeric form. Please present Table 2 as 'simulated user satisfaction' and either add external validation (real-user study, logged behavior, or a different simulation paradigm) or carefully hedge the abstract's satisfaction claim.","section":"§5.1.2; §4.1.1; §4.3"},{"comment":"The comparison is confounded by unequal compute. Self-EvolveRec uses ~25 min/iteration vs ~6 min (AlphaEvolve) and ~14 min (DeepEvolve) (Table 8). The paper argues fewer iterations to peak, but does not report total compute or equalize the LLM-inference/training budget. The observed gains may partly reflect greater computational cost rather than directional feedback. Please add an equal-compute comparison (same wall-clock or same number of LLM calls) or an analysis isolating the effect of the directional signal.","section":"§5.3.5; App. F.1.2; Table 4"},{"comment":"The codebase-quality evaluation uses GPT-5 as LLM-as-a-judge—the same model family that generated the evolved code—without human validation or inter-rater agreement. The judge's criteria overlap with the instructions used in the evolution loop, creating a self-evaluation bias. The claim that Self-EvolveRec produces higher 'Creativity'/'Insight'/'Personalization' would be stronger with human expert ratings on a sample of evolved codebases, or at least an independent judge model and reported agreement.","section":"§5.1.3; App. G.2"}],"minor_comments":[{"comment":"Typo: 'Amzaon' should be 'Amazon'.","section":"§5 Datasets"},{"comment":"The caption contains stray Korean text ('진짜진짜진짜진짜 최종') that should be removed.","section":"Figure 5 caption"},{"comment":"Satisfaction results are reported only for SASRec and NCF seeds; please justify the omission of NGCF and MoRec, or report them.","section":"Table 2"},{"comment":"The phrase 'deterministic improvements' overstates the determinism of an LLM-based pipeline; consider 'consistent' or 'robust' improvements.","section":"§5.3.5"},{"comment":"Equation (1) presents a bi-level optimization problem that the evolutionary search does not actually solve; the text should clarify that this is a goal specification, not the optimization procedure.","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has several artifacts of an unfinished submission (ACM template with 2018 dates, 'Received 20 February 2007', stray Korean text in Fig. 5). The technical concerns above are the substantive ones; I do not see evidence of misconduct. The empirical breadth is considerable, and with statistical tests and a more careful satisfaction framing, the paper could be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is a genuine combination: an LLM user simulator supplies qualitative critiques, a model diagnosis tool supplies quantitative probes, and both co-evolve inside an open-ended code evolution loop. The standard-metric results support the core idea reasonably well. Across four datasets and four seed recommenders, the evolved pipelines beat the AlphaEvolve/DeepEvolve-style baselines, and the ablations show each component contributes. The diagnosis-tool co-evolution is the most interesting piece. The injected-deficiency case studies—removing positional embeddings and seeing the new 'Swap Sensitivity' metric catch it—give me real confidence that the co-evolved diagnostics measure what they claim. The paper is also honest about substituting OpenEvolve for AlphaEvolve because the official code is unavailable; that's a fair workaround.\n\nThe soft spots are about evidence strength, not about the method's coherence. The prose says 'significantly outperforms,' but there are no error bars or significance tests for a stochastic LLM process. Some reported gaps are small, and the ordering could shift with different seeds. That's fixable, but it should be fixed before publication.\n\nThe bigger issue is the user-satisfaction half of the headline. The satisfaction evaluation uses Agent4Rec and PUB, the same family of LLM user simulators that drive the feedback loop. So the loop is being judged by the same kind of oracle that guided it. The DIAG co-evolution also transduces SIM critiques into numeric probes; it doesn't independently verify the SIM, it operationalizes it. That makes the satisfaction claim an in-simulation claim, not an external validation. It doesn't sink the paper—the NDCG/HR results are independent and consistent—but the abstract's user-satisfaction claim should be softened or backed by a real-user study, or at least a clear argument for why the simulator's preferences should transfer.\n\nMinor: the LLM-as-a-judge code-quality scores are subjective, but they're labeled as such and are secondary.\n\nOverall: this paper deserves a serious referee. The mechanism is clearly articulated, the experiment suite is broad, and the case studies add credibility. The revision needs statistical rigor and a more honest framing of the satisfaction results. I'd bring it to the reading group and would cite it in my own work on LLM-agent pipeline optimization.","headline":"Real combination of user-simulator critique and co-evolving diagnostics inside LLM code evolution; standard-metric gains are credible, but the user-satisfaction claim is in-simulation, not independently validated.","tokens_in":31286,"tokens_out":3594,"would_cite":true,"duration_ms":31203,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM agents evolve recommenders using critiques, not just scores","keywords":["recommender systems","LLM-driven code evolution","directional feedback","user simulation","model diagnosis","co-evolution","agentic AI","self-evolving agents"],"falsifier":"Run Self-EvolveRec with an inverted user simulator whose critiques deliberately reward the opposite of genuine preference, and check whether standard NDCG and HR still rise; or deploy two evolved pipelines, one guided by simulator critiques and one by scalar metrics, in an online experiment comparing real-user retention and satisfaction. If simulated satisfaction diverges from real-user satisfaction, the directional loop's core claim fails.","tokens_in":30266,"feed_emoji":"🎯","tokens_out":3102,"duration_ms":31524,"temperature":0.7,"pith_summary":"The paper argues that LLM-driven code evolution for recommender systems fails when guided only by scalar accuracy metrics, because NDCG collapses many distinct failure modes—popularity bias, category mismatch, recency blindness—into a single number. Self-EvolveRec adds two feedback channels: a User Simulator that critiques recommendation lists in plain language from persona-conditioned viewpoints, and a Model Diagnosis Tool that probes the model's internals for quantitative confirmation. The two channels are cross-checked, and the diagnosis tool itself evolves alongside the model so its probes match new architecture components. Experiments across four datasets and four seed recommenders report consistent gains over NAS and scalar-driven LLM evolution baselines, plus higher simulated user satisfaction.","feed_headline":"Critiques and probes beat raw scores in recommender evolution","feed_subtitle":"LLM-driven pipeline grows recommenders by diagnosing why users are unhappy, with fewer wasted iterations.","key_machinery":"The directional feedback loop: a User Simulator (SIM) generates persona-conditioned natural-language critiques, while a Model Diagnosis Tool (DIAG) computes measurable internal probes such as embedding collapse (mean pairwise cosine similarity of item embeddings) and ranking margin (logit gap between ground-truth and negative items). An LLM planner turns both feedback streams into a development report, and a coding agent applies targeted code edits. The co-evolution step re-derives DIAG after each architecture change, ensuring the probes can verify newly added components and translate the simulator's qualitative complaints into concrete numerical signals.","core_discovery":"The central claim is that pairing qualitative user critiques with quantitative internal probes lets an LLM coding agent pinpoint root causes of failure and make targeted code changes, producing deterministic improvements in the recommendation pipeline rather than undirected trial-and-error. The Diagnosis Tool-Model Co-Evolution mechanism keeps verification criteria aligned with a shifting codebase: when the simulator reports a new failure mode such as order-insensitivity, the diagnostic tool autonomously generates corresponding probes, like swapping the last two items in a sequence and measuring ranking shifts. This cross-verification loop is what distinguishes the framework from scalar-metr","pith_inferences":["The critical open question is transfer to real users: satisfaction evidence uses simulated agents of the same kind that generate the training feedback, so a real-user A/B test would be the decisive next check.","Because the diagnostic tool makes feedback objectives explicit, its probes could double as auditable constraint signals to monitor or restrict evolved code.","Per-iteration cost is substantial; a promising efficiency test is predicting metric outcomes directly from proposed code diffs instead of retraining and re-evaluating every variant.","If the simulator's critique distribution is miscalibrated, the loop could overfit to simulated personas; injecting a known critique bias and measuring metric drift would reveal sensitivity."],"forward_implications":["Recommender optimization can escape fixed NAS operator pools: the search target becomes whole codebases, including losses, sampling logic, and encoders.","Evolution becomes cause-directed: each code change is traceable to a diagnosed failure mode, reducing wasteful exploration iterations.","Performance gains extend to user-centric metrics such as view ratio, satisfaction score, and browsing depth, not just ranking accuracy.","The verification mechanism stays valid as the architecture drifts, because the diagnostic tool co-evolves with the model.","The same pipeline works from a random recommender up to a strong ensemble, suggesting applicability across the service lifecycle."],"fun_headline_variants":["Self-evolving recommenders learn from critiques and probes","LLM co-evolves recommenders with directional feedback","Diagnosis-driven evolution beats scalar metrics in recommenders","User simulators and internal probes guide recommender evolution","From raw scores to root causes: recommender evolution reboot"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that LLM-generated critiques from persona-conditioned simulated users reflect real user preferences closely enough that optimizing for them improves real satisfaction and accuracy; the paper validates satisfaction with the same class of simulators, so this fidelity is assumed rather than demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["Self-evolving recommenders learn from critiques and probes","LLM co-evolves recommenders with directional feedback","Diagnosis-driven evolution beats scalar metrics in recommenders","User simulators and internal probes guide recommender evolution","From raw scores to root causes: recommender evolution reboot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000755,"raw_usage":{"total_tokens":3166,"prompt_tokens":685,"completion_tokens":2481,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":2404}},"tokens_in":429,"tokens_out":2481,"duration_ms":15329,"temperature":1.0,"reasoning_tokens":2404,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:45:16.024011+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Self-EvolveRec with an inverted user simulator whose critiques deliberately reward the opposite of genuine preference, and check whether standard NDCG and HR still rise; or deploy two evolved pipelines, one guided by simulator critiques and one by scalar metrics, in an online experiment comparing real-user retention and satisfaction. If simulated satisfaction diverges from real-user satisfaction, the directional loop's core claim fails.","supporting_citations":[],"review_version":1}