{"id":"88f6cb01-2c5a-4bd2-b83c-683f681d2ea0","arxiv_id":"2505.19623","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"AgentRecBench is a public text-based benchmark for LLM recommendation agents, but its headline claim of agent superiority is undercut by its own tables.","lead":"AgentRecBench is a new benchmark that tests how well LLM-based agents recommend items by letting them query a textual simulation of user, item, and review data from Yelp, GoodReads, and Amazon. It provides the shared testbed this field needs, with a public leaderboard, but the paper's own results weaken its claim that agents are superior.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Traditional baselines in Tables 2-3 are not trained models: 15.0 equals random-chance HR@N, admitted as 'mean prediction' in Sec 5.1, yet Table 4 shows the same models train well - so the agentic-superiority claim is unsupported.","rationale":"The reader's verdict is REJECT, and my stress-test supports that rejection, so no verdict change is needed. The reader's weakest assumption was that the chance-level MF/LightGCN results were due to an undersized training subset on the classic task; my analysis identifies a more fundamental problem: the 15.0 entries are not trained model outputs at all, but are explicitly described in Sec 5.1 as the 'mean prediction' reported as a reference point, which is probabilistically equivalent to random ranking. This is internally contradicted by Table 4, where the same model families are trained and reach 65.9-68.9 on Yelp short-term, demonstrating that the models can learn when given the appropriate protocol. The absence of valid traditional baselines directly undermines the abstract's superiority claim and the derived design guidelines, which are the paper's core contributions. I credit the authors for building and publicly releasing an interactive benchmark environment and for validating it through the AgentSociety Challenge; that is a genuine artifact. However, a benchmark environment is not the same as a demonstrated comparative result. Appendix A.2 itself admits that 'more traditional and deep learning-based baselines' are needed, which is a fair limitation statement but does not repair the current tables. I also note the absence of error bars and the anomalously low Yelp agent scores (all below chance in Tables 2 and 3), which are additional reasons to distrust the headline comparison. My concrete test isolates the decisive check: retrain the traditional models on the classic-task subset with a proper protocol and see whether the 15.0 row moves. If it does, the central claim fails; if it remains at 15.0 after correct training, the sparsity explanation would have some support. Until that test is run, the claim that agentic systems demonstrate superiority is unverified, and the REJECT verdict stands.","tokens_in":13374,"tokens_out":4191,"duration_ms":40441,"concrete_test":"Retrain MF and LightGCN on the classic-task training subset using the same standard protocol that produced the Table 4 evolving-interest results (full negative sampling, hyperparameter tuning, early stopping), then evaluate with the same 20-candidate HR@N protocol. If either model's Amazon/GoodReads/Yelp HR@N substantially exceeds 15.0, the chance-level row in Tables 2-3 is a training/evaluation artifact and the agentic-superiority claim is unsupported. A minimal variant: run the same experiment but replace the 'mean prediction' ranking with the trained model's scores; if HR@N moves away from 15.0, the baseline was not a real comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison is invalid because the 'Traditional RS' and 'DL-based RS' rows in Tables 2 and 3 are not trained model outputs. Every entry is exactly 15.0, which is the expected HR@N for uniformly random ranking of 20 candidates (HR@1=5%, HR@3=15%, HR@5=25%; average=15%). Section 5.1 explicitly states that due to high sparsity 'these models are unable to effectively learn meaningful patterns' and 'we report the mean prediction as a reference point for comparison.' Reporting the mean prediction is equivalent to ranking by a constant score, i.e., a no-signal random baseline. Consequently, the headline comparison of agentic systems against MF and LightGCN in the classic and cold-start scenarios does not exist. This is not a harmless detail: the abstract and Contribution 3 claim that 'our findings demonstrate the superiority of agentic systems' and yield actionable design guidelines. The only evidence for that claim in the main tables is a comparison against a dummy random baseline. The contradiction with Table 4 makes the explanation in Sec 5.1 untenable: the same MF and LightGCN models, trained on the same datasets, achieve HR@N of 32.3-59.1 (Amazon), 17.7-22.3 (GoodReads), and 31.6-68.9 (Yelp) in the evolving-interest scenario. Thus the models can learn when given a proper training setup; the chance-level classic-task results are an artifact of the protocol, not of data sparsity. Additionally, on Yelp all agents score 3.7-7.7 in Table 2 and 0.3-4.3 in Table 3, well below the chance-level 15.0 baseline. So even relative to chance, the agentic systems perform poorly on one of the three benchmark domains. The paper does not reconcile this, and no variance or error bars are provided, so differences among agents (e.g., 54.0 vs 44.0 in Table 2) cannot be assessed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AgentRecBench, a benchmark for evaluating LLM-agent-based personalized recommender systems. It provides an interactive textual environment built from Yelp, GoodReads, and Amazon data, with three evaluation scenarios (classic, evolving-interest, and cold-start) and a modular agent framework. The authors evaluate eight agentic systems plus two conventional baselines (MF and LightGCN), and claim that agentic systems are superior and that the results yield actionable design guidelines for constructing recommendation agents.","tokens_in":13689,"tokens_out":3503,"duration_ms":36381,"significance":"If the empirical claims were valid, AgentRecBench would be a useful community resource: the environment design, dataset integration, scenario taxonomy, and the open challenge are valuable contributions, and the released benchmark with leaderboard could foster reproducible research. However, the central comparative claims are not supported by the evidence presented. The conventional baselines in two of the three scenarios are reported as a constant 15.0 HR@N, which is exactly the random-chance expectation, while the same models attain strong results in the third scenario. The claimed 'superiority of agentic systems' therefore rests on comparison against a random baseline in the classic and cold-start tasks, and is actively contradicted by the evolving-interest results. The design guidelines are also distilled from the winners of a challenge run on this same benchmark, creating an in-sample circularity that undermines their generality.","major_comments":[{"comment":"The 'Traditional RS' (MF) and 'DL-based RS' (LightGCN) rows in Tables 2 and 3 report a constant value of 15.0 for every dataset and scenario. This value is precisely the expected Hit Rate for a uniformly random ranking over 20 candidate items: HR@1=5%, HR@3=15%, HR@5=25%, and their average is 15%. Section 5.1 confirms that these models 'are unable to effectively learn meaningful patterns' and that 'we report the mean prediction as a reference point for comparison.' Ranking by a constant mean prediction is equivalent to a no-signal random baseline. Consequently, the paper's headline claim that agentic systems are superior to traditional methods in the classic and cold-start scenarios is unsupported: those comparisons pit agents against a dummy baseline, not against trained recommendation models.","section":"Tables 2 and 3; Section 5.1"},{"comment":"The same MF and LightGCN models that yield 15.0 in the classic and cold-start tasks achieve HR@N values of 32.3–59.1 on Amazon, 17.7–22.3 on GoodReads, and 31.6–68.9 on Yelp in the evolving-interest scenario (Table 4). This directly contradicts the claim that high data sparsity prevents these models from learning on the benchmark datasets. The correct interpretation is that the classic and cold-start tasks were constructed without a usable training set for collaborative filtering methods, so the chance-level results are an artifact of the evaluation protocol, not evidence of inherent advantages of agentic systems. The authors must either repair the baseline training/evaluation setup for those scenarios or refrain from making comparative superiority claims based on the current tables.","section":"Table 4 vs. Tables 2 and 3; Section 5.1"},{"comment":"The three agents identified as 'superior' (Baseline666, DummyAgent, and RecHackers) are, respectively, the first-, third-, and second-place solutions from the AgentSociety Challenge [13], which was run using this same benchmark and its leaderboard. The design guidelines in Appendix A.4 are distilled from these agents' workflows. Because these agents were selected by competing on this exact benchmark, the guidelines may reflect benchmark-specific overfitting rather than general principles for agentic recommender systems. No out-of-sample validation is provided to show that the guidelines transfer to agents not developed on this benchmark, so the 'actionable design guidelines' claim (Contribution 3) is not established.","section":"Section 4.3 and Appendix A.4"},{"comment":"The limitations section states that the authors 'plan to incorporate more traditional and deep learning-based baselines for more thorough comparative analysis.' This admission is in tension with the paper's claim in the Introduction and Abstract of being 'the first comprehensive benchmark' that 'systematically evaluates both emerging agentic recommender systems and traditional recommendation methods.' A benchmark whose only two traditional baselines fail to train in two of the three scenarios cannot currently substantiate the 'comprehensive' and 'superiority' claims. The benchmark may still be a useful resource, but the present evaluation is incomplete in a way that directly affects the paper's main conclusions.","section":"Appendix A.2"}],"minor_comments":[{"comment":"The subsection heading 'Envaluation Metric' contains a typo and should read 'Evaluation Metric.'","section":"Section 4.4"},{"comment":"The phrase 'we report the mean prediction as a reference point' is not defined precisely; the authors should state explicitly what 'mean prediction' means operationally (e.g., ranking all items by the same predicted score, or averaging the rating predictions).","section":"Section 5.1"},{"comment":"The supplementary tables report results for DeepSeek-V3 and GPT-4o-mini, but no statistical significance tests or variance estimates are provided for any of the tables; given the small differences between many agents (e.g., 44.7 vs. 45.6 in Table 3), such tests or repeated-run statistics would be important for the reliability of the ranking conclusions.","section":"Tables 5 and 6"},{"comment":"In the Yelp cold-start columns, many agentic methods score at or below the chance level of 15.0 (e.g., 0.3–4.3), which may indicate a data or evaluation anomaly in that scenario; the authors should discuss this pattern, as it suggests that the Yelp cold-start task may not be functioning as intended.","section":"Table 3"},{"comment":"Figures 1 and 2 are difficult to read in the preprint resolution; the text in the figures is small and the control-flow arrows are hard to follow, reducing the clarity of the environment design described in Section 3.","section":"Figures 1 and 2"}],"recommendation":"reject","confidential_remarks":"The paper is probably best viewed as a systems/resources contribution, and the open challenge and released environment are useful. However, the scientific claims of agentic superiority and design guidelines are not supported by the experiments as presented. The constant 15.0 baseline issue is a fundamental correctness problem in the main tables, and the contradiction with Table 4 means that simply adding a stronger baseline section is unlikely to fix the core conclusions. The circularity of deriving guidelines from the same challenge winners is also a structural concern. Given these load-bearing issues, I cannot recommend publication in its current form, and in my view the flaws are too deep to be resolved by a revision within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about AgentRecBench. One: the interactive textual environment with three task scenarios is a real, potentially useful artifact, and the open challenge that ran on it (295 teams, ~1,400 submissions) shows it's already getting traction. Two: the headline finding—that agentic systems beat traditional recommenders—is not supported by the paper's own tables. The MF and LightGCN rows in Tables 2 and 3 are all exactly 15.0, which is the expected HR@N for a uniform random ranking of 20 candidates. Section 5.1 admits these models \"are unable to effectively learn meaningful patterns\" on the training subset and that they report the mean prediction as a reference point. That is a dummy baseline, not a trained model.\n\nWhat the paper does well: the three scenarios (classic, cold-start, evolving-interest) are sensible, the environment offers standard query interfaces with dynamic time/item filtering, and the authors are transparent in Appendix A.2 that more baselines and multimodal data are needed. The case studies of the winning agent prompts are also informative for practitioners.\n\nThe soft spots, in order of severity. First, the contradiction with Table 4 is fatal to the explanation. The same MF and LightGCN models, on the same datasets, produce 17.7–68.9 HR@N in the evolving-interest tasks. So they can learn when the training setup is reasonable; the chance-level classic-task results are an artifact of the protocol, not inherent data sparsity. Second, on Yelp every agent in Tables 2 and 3 scores below the 15.0 random baseline (e.g., 3.7–7.7 in Table 2). The paper never acknowledges this. Third, the three \"superior\" agents (Baseline666, DummyAgent, RecHackers) are the winner, runner-up, and third-place of the AgentSociety Challenge run on this same benchmark, so the design guidelines distilled from their workflows are in-sample; they may not transfer. Fourth, no error bars or variance are reported, so small differences between agents (e.g., 54.0 vs 44.0) are uninterpretable.\n\nWho is this for: researchers building or evaluating agentic recommender benchmarks. The environment is worth reusing, but the paper needs a serious revision: the authors must actually train the traditional baselines in all scenarios, report variance, and either drop the superiority claim or support it with real comparisons. The current version should not be accepted as is, but it deserves peer-review attention because fixing the evaluation would make it a useful community resource.","headline":"The benchmark environment is a solid artifact, but the central comparison is broken: the traditional baselines in the classic/cold-start tables are random-chance constant predictions, not trained models.","tokens_in":14399,"tokens_out":2649,"would_cite":false,"duration_ms":25023,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgentRecBench is a new benchmark that reports LLM-agent recommender systems outperform traditional methods and yields design guidelines for building them.","keywords":["agentic recommender systems","LLM agents","recommendation benchmark","textual environment simulator","cold-start recommendation","evolving interests","hit rate"],"falsifier":"Train MF and LightGCN on the full classic-task training data (or a better-tuned subset) and measure HR@N on the same 20-candidate test set; if their scores rise well above the reported 15.0 chance level, the claim of agentic superiority in the classic scenario is empirically wrong as stated.","tokens_in":13094,"feed_emoji":"🤖","tokens_out":5628,"duration_ms":44201,"temperature":0.7,"pith_summary":"AgentRecBench is a proposed benchmark for LLM-based agentic recommender systems, built on a textual interaction simulator over Yelp, GoodReads, and Amazon data. The paper claims to provide the first comprehensive, standardized evaluation that compares eight agentic systems against traditional methods across classic, evolving-interest, and cold-start scenarios. Its central finding is that well-designed agentic systems, especially those using platform-specific item and review feature engineering, outperform simpler agents and classical baselines, and it distills this into actionable design guidelines. A sympathetic reader would care because agentic recommenders are a fast-growing approach that currently lacks a common testbed, so a validated benchmark would let the field measure progress and transfer lessons across methods.","feed_headline":"LLM-agent recommenders outrank classic models in new benchmark","feed_subtitle":"AgentRecBench offers a standard testbed and design rules from top agents to speed up recommender research.","key_machinery":"The load-bearing object is the interactive textual environment simulator, which models a platform as a User-Review-Item network and exposes a standardized query function Query(Type, SortMethod, Formation) that agents call to retrieve structured or textual data. A two-layer dynamic data visibility control (scenario-level filters over time and item properties, task-level target user and ground truth) makes the same environment serve classic, evolving-interest, and cold-start evaluations. This machinery is what makes fair, reproducible comparison possible: every tested agent sees the same queryable world and the same 20-candidate ranking tasks, scored by Hit Rate@N. The modular agent framework (planning, reasoning, tool use, memory) is the accompanying toolkit for building the agents being compared.","core_discovery":"On the paper's own terms, the central discovery is that AgentRecBench constitutes a reliable and comprehensive benchmark for agentic recommender systems, and that on it the best-performing agents (Baseline666, DummyAgent, and RecHackers) consistently surpass simpler agent designs and classical methods across domains and scenarios. The paper establishes a unified textual interaction environment with dynamic data visibility control, and shows through an open challenge with 295 teams that the benchmark is usable and results are reproducible. From the performance differences it derives design guidelines: effective agent workflows combine user history, candidate item details, and platform-specific features; they extract representative domain-adapted item attributes; and they prioritize information-rich reviews. The paper asserts these findings demonstrate the superiority of agentic systems and provide actionable guidance for building them.","pith_inferences":["We infer that training MF and LightGCN on the full classic-task dataset rather than a small subset could change the headline comparison, since the paper itself reports these models reach substantive accuracy in the evolving-interest scenario, suggesting the classic-task chance-level results reflect the training subset, not the methods.","The winning agents' heavy reliance on platform-specific features implies that 'general' agentic recommendation may actually be domain-customized ranking with LLM glue, a nuance worth stating for practitioners.","One could extend the benchmark to include traditional models trained on the full data, as well as more deep-learning baselines, and re-run the same three scenarios to test the generality of the claimed superiority.","The benchmark's textual-only environment leaves open whether the design guidelines transfer to multimodal platforms (e.g., image-based recommendations), which the paper itself lists as future work."],"forward_implications":["If AgentRecBench is right, agentic recommender systems can be compared on a common, reproducible testbed, ending the field's reliance on ad-hoc evaluations.","The design guidelines, such as platform-aware item feature extraction and review-side engineering, give concrete starting points for building new recommendation agents, not just tuning prompts.","The three-scenario design (classic, evolving-interest, cold-start) means an agent's robustness to distribution shift and data sparsity becomes measurable alongside raw accuracy.","The benchmark's validation through a 295-team open challenge suggests the environment is usable by outside researchers and its leaderboard can track progress over time."],"supporting_citations":[{"why":"Supplies the Matrix Factorization baseline used as a traditional recommender system in the comparisons.","marker":"[5]"},{"why":"Supplies the LightGCN graph-based deep learning baseline compared against agentic systems.","marker":"[8]"},{"why":"Source of the top three winning agents (Baseline666, RecHackers, DummyAgent) and the open challenge that validated the benchmark.","marker":"[13]"},{"why":"Provides Agent4Rec, an existing agentic recommender system included as a baseline.","marker":"[15]"},{"why":"Provides the memory mechanism used in the MemoryAgent and CoTMemAgent baselines.","marker":"[19]"},{"why":"Provides the chain-of-thought prompting method used in the CoTAgent baseline.","marker":"[26]"}],"fun_headline_variants":["New benchmark ranks LLM agent recommenders above classics","AgentRecBench: standard testbed for agentic recommender research","Benchmark shows LLM agents beat classical recommenders","Agentic recommenders outclass classic in first benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the traditional baselines, trained on a small subset of classic-task data and unable to learn there, fairly represent what classical recommenders can do in this benchmark.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark ranks LLM agent recommenders above classics","AgentRecBench: standard testbed for agentic recommender research","Benchmark shows LLM agents beat classical recommenders","Agentic recommenders outclass classic in first benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000669,"raw_usage":{"total_tokens":3059,"prompt_tokens":960,"completion_tokens":2099,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":2033}},"tokens_in":576,"tokens_out":2099,"duration_ms":14052,"temperature":1.0,"reasoning_tokens":2033,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:10:37.755628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MF and LightGCN on the full classic-task training data (or a better-tuned subset) and measure HR@N on the same 20-candidate test set; if their scores rise well above the reported 15.0 chance level, the claim of agentic superiority in the classic scenario is empirically wrong as stated.","supporting_citations":[{"cited_title":"On generative agents in recommendation","cited_arxiv_id":null,"evidence_quote":"Provides Agent4Rec, an existing agentic recommender system included as a baseline."},{"cited_title":"Generative agents: Interactive simulacra of human behavior","cited_arxiv_id":null,"evidence_quote":"Provides the memory mechanism used in the MemoryAgent and CoTMemAgent baselines."}],"review_version":1}