{"id":"8e16c829-879a-401c-8ea6-3f8c8a531457","arxiv_id":"2608.08417","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A why-what-how-who communication skill bank, with personalized routing and failure-driven skill evolution, improves Hit@1 for agentic recommender systems in offline tests.","lead":"AgentCom is a framework that lets AI recommender agents consult simulated advisor agents before choosing what to recommend. The paper reports that structuring this communication with reusable why, what, how, and who skills improves how often the system picks the user's actual next item across several recommender backbones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No error bars, seeds, or significance tests: the headline 'consistently improves' across all 15 cells rests on single-run Hit@1 differences, several of which are tiny (e.g., MemRec LastFM +0.0048).","rationale":"The paper proposes a plausible framework and the experiments are extensive, but the headline empirical claim is stronger than the evidence supports. The most load-bearing assumption is that the reported Hit@1 improvements are stable effects of AgentCom. All components are stochastic LLM calls, and the paper gives no variance information, so even the small gains used to support 'consistent' improvement could be noise. I checked whether the candidate-set protocol is a deeper flaw: because the candidate set is ground truth plus the top 19 non-ground-truth items, the original SASRec baseline's top-1 under its own scores would be unchanged if it were restricted to that candidate set—if the ground truth is not the global top item, the highest-scoring item in the candidate set is the global top non-ground-truth item. Thus the baseline comparison is not as asymmetric as it first appears. The absence of a generic LLM-reranking baseline remains relevant, but the decisive gap is statistical: without repeated runs, confidence intervals, or significance tests, the claim that AgentCom improves every backbone on every dataset cannot be adjudicated. The reader's weakest_assumption identified the same issue, so I agree with the conditional verdict and recommend no change.","tokens_in":15407,"tokens_out":7555,"duration_ms":84603,"concrete_test":"Rerun the full Table 2 experiment at least 5 times per condition with different random seeds (or temperature/prompt variations) while keeping candidate sets and protocol fixed; report mean and standard deviation over seeds. Compute paired bootstrap or permutation confidence intervals over test users for each backbone-dataset pair, comparing AgentCom-enhanced against the original. The concern is settled if all 15 pairwise differences remain positive and the 95% confidence intervals for the smallest gains (MemRec on LastFM and Epinions, iAgent on Epinions) exclude zero. Also run a generic LLM-reranking baseline that consumes the same advisor evidence budget without the why-what-how-who routing and evolution; if the generic baseline matches Table 2, the communication-skill structure is not what drives the gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AgentCom consistently improves recommendation performance across every backbone and dataset. For that claim to hold, each positive difference in Table 2 must reflect a real effect of the method rather than run-to-run variation in stochastic LLM outputs. Section 4.2 reports a single run per condition, with no standard deviations, confidence intervals, significance tests, seeds, or temperature settings. The UserAgents, advisor agents, router, and failure analyzer are all LLM-based, so the pipeline is inherently stochastic. Several reported gains are small: MemRec improves by +0.0048 on LastFM (1.8% relative) and +0.0058 on Epinions (2.4% relative), while iAgent improves by only +0.0154 on Epinions. With typical test-set sizes, these differences could easily be within the noise of one run, especially because the underlying LLM can flip a small number of borderline decisions between runs. The candidate-set protocol itself (Section 4.1.4) is not the primary problem: because the candidate set consists of the ground-truth item plus the 19 highest-ranked non-ground-truth items, restricting the original SASRec baseline to that set would not change its Hit@1, so the baseline comparison is not obviously biased. The load-bearing gap is that no uncertainty quantification is provided, and no generic LLM-reranking baseline is included to rule out the possibility that the improvements come from LLM candidate evaluation rather than the why-what-how-who skill structure. Without repeated runs or significance testing, the 'consistently improves' statement is not supported by the evidence as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AgentCom, a framework that inserts a personalized communication-skill layer into agentic recommender systems. AgentCom maintains a shared why--what--how--who skill bank, routes each user and recommendation context through a personalized communication path, and evolves the bank from training-time failures via skill refinement, skill generation, and routing correction. Experiments on LastFM, Epinions, and LibraryThing compare five backbones (SASRec, GBSR, AFL, iAgent, MemRec) with and without AgentCom, reporting Hit@1 in Table 2, plus ablations, hyperparameter analysis, a case study, and an LLM cost analysis. The central claim is that AgentCom consistently improves recommendation performance across traditional, social, and agentic recommenders.","tokens_in":15731,"tokens_out":4623,"duration_ms":51697,"significance":"If the empirical claim is supported, the paper makes a useful conceptual contribution: framing advisor-agent communication as reusable, hierarchically organized skills with a why--what--how--who decomposition, and complementing it with failure-driven skill evolution. This is a plausible and timely direction for agentic recommender systems, and the shared-bank, personalized-routing design is a sensible way to make communication adaptive without per-user prompt engineering. The paper also ships a complete algorithmic description and cost analysis, which strengthens reproducibility. However, the main claim rests on single-run Hit@1 numbers without uncertainty quantification and on comparisons that lack a non-communication LLM reranking baseline, so the current evidence does not yet establish that the observed improvements are due to the proposed skill structure.","major_comments":[{"comment":"The central claim that AgentCom \"consistently improves\" performance across all backbones and datasets is supported only by single-run Hit@1 values. No standard deviations, confidence intervals, significance tests, random seeds, or sampling temperatures are reported, despite the pipeline being composed of stochastic LLM components (UserAgents, advisor agents, router, and failure analyzer). Several differences are small in absolute terms, for example MemRec on LastFM improves from 0.2648 to 0.2696 (+0.0048) and on Epinions from 0.2427 to 0.2485 (+0.0058); these could easily be within run-to-run noise. To support the headline claim, the authors should run multiple independent seeds, report means and spreads, and perform paired significance tests such as McNemar's test or a bootstrap over users.","section":"Section 4.2, Table 2"},{"comment":"There is no baseline that reranks the same 20-item candidate set with the same underlying LLM but without any communication-skill structure. The \"w/o all\" ablation replaces the structured skill with generic dialogue, but it is not described precisely: it must be stated whether generic dialogue still includes advisor agents and, if so, how advisors are selected. Without a pure LLM-reranking baseline, the measured gains could be attributed to the LLM's ability to evaluate candidates rather than to the why--what--how--who skill decomposition. The authors should add a no-communication LLM reranker and a generic single-advisor dialogue variant to isolate the contribution of the skill structure.","section":"Section 4.1.4 and ablation in Section 4.3"},{"comment":"The hyperparameter conclusions about Kmax and G are based on single-run Hit@1 differences as small as 0.0039 and 0.0048, with no error bars or significance tests. The claim that increasing G reduces performance and that Kmax = 3 is the right operating point should be supported by repeated runs or confidence intervals, especially because the differences are comparable in magnitude to the small gains in Table 2.","section":"Section 4.4, Figure 4"}],"minor_comments":[{"comment":"The evaluation protocol always inserts the ground-truth item into the 20-item candidate set. This is fair for within-backbone comparisons because both the original and AgentCom-enhanced variants operate on the same candidate set, but it means the absolute Hit@1 values in Table 2 are not directly comparable to standard top-k metrics in the literature; the authors should state this limitation explicitly.","section":"Section 4.1.4"},{"comment":"The caption says that AgentCom-enhanced variants are \"highlighted in blue, with their results shown in bold,\" but the table as presented has no visible blue highlighting or bold formatting; this is a typesetting issue that should be corrected.","section":"Table 2 caption"},{"comment":"The ablation figure would be more informative if accompanied by a table of exact Hit@1 values, since the current text reports only qualitative decreases and the figure is hard to read at low resolution.","section":"Section 4.3, Figure 3"},{"comment":"The RankLLM operation is central to routing, but the paper does not specify the prompt template, the number of nodes given to the router, or the temperature used; adding these details would substantially improve reproducibility.","section":"Section 3.2.1, Eq. (12)"},{"comment":"The notation K_{u,t} is introduced as the number of communication rounds and later reused in Algorithm 1 and the hyperparameter section as Kmax; the relation between K_{u,t} and Kmax should be stated explicitly in Section 2.2.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is on a timely topic and the conceptual framework is interesting, but the empirical support for the main claim is not yet convincing. The missing error bars, seeds, and non-communication baseline are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper's contribution is real but modest: AgentCom gives a reusable why-what-how-who skill bank for letting a recommender's UserAgents solicit advisor evidence, with personalized routing and failure-driven evolution. I haven't seen that exact combination in the cited literature, and the ablation suggests each layer earns its keep. The cost analysis in Appendix C is a nice touch—most agentic recommender papers ignore API cost.\n\nThe biggest soft spot is exactly what the stress-test says: every claim of improvement rests on one run per condition. No seeds, no variance, no significance tests. Some gains are tiny (MemRec on LastFM +0.0048 absolute, 1.8% relative), and the LLM pipeline is stochastic to its core. The 'consistently improves' line in the abstract is stronger than the evidence as reported. I'd also want a non-communication LLM reranking baseline in the main table; without it, you can't tell whether the gains come from the skill structure or simply from adding an LLM over the candidate set. The candidate-set protocol itself is fine—ground truth is always in the 20 items, and the baselines see the same set, so the comparison isn't biased. The circularity worry is also low; the skill bank is fixed for inference and evolution uses training labels, which is standard supervised practice.\n\nThe framework is coherent and the authors are addressing a real problem. It deserves peer review, but the referee should insist on repeated runs with variance, significance tests, and release of the skill-bank definitions and prompts. As is, I'd be cautious citing the quantitative results, though the framework is worth knowing about for anyone working on agentic recommendation or multi-agent LLM communication.","headline":"AgentCom is a coherent communication-skill layer for agentic recommenders, but its headline 'consistently improves' rests entirely on single-run Hit@1 numbers with no error bars, so treat the gains as unverified until repeated runs and significance tests appear.","tokens_in":16252,"tokens_out":3079,"would_cite":false,"duration_ms":29577,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgentCom claims that adding personalized why-what-how-who communication skills to UserAgents improves recommendation accuracy across traditional, social, and agentic recommenders.","keywords":["agentic recommender systems","user agents","advisor agents","communication skills","why-what-how-who skill bank","personalized skill routing","failure-driven skill evolution","large language models"],"falsifier":"Re-run the Table 2 protocol with at least ten random seeds and report mean, standard deviation, and confidence intervals; if the AgentCom-over-baseline gaps for SASRec and GBSR overlap zero, or if removing the guaranteed ground-truth item from the 20-candidate set eliminates the advantage, the central claim is not supported.","tokens_in":15220,"feed_emoji":"💬","tokens_out":7002,"duration_ms":70336,"temperature":0.7,"pith_summary":"This paper argues that UserAgents—large-language-model simulations of a person used to test candidate items before a recommendation is delivered—choose better when they can consult other users' agents as advisors, and that the consultation has to be matched to the user's current decision problem instead of following a fixed script. To make that work, the authors build AgentCom, a layer that sits on top of any recommender and organizes advisor communication as reusable skills along four questions: why advice is needed, what information to request, how advisors should interact, and who should advise. The paper reports consistent Hit@1 improvements when AgentCom is added to all five backbones tested on three datasets, including raising the sequential model SASRec's LastFM accuracy from 0.0155 to 0.2019 and improving the strongest agentic baseline MemRec. If the measurements hold, cross-user agent communication becomes a general plug-in that can upgrade existing recommenders without retraining them.","feed_headline":"Consulting other simulated users raises recommendation accuracy","feed_subtitle":"AgentCom's why-what-how-who skill bank lifts Hit@1 for every backbone, from SASRec to MemRec, on three datasets.","key_machinery":"The load-bearing object is the why–what–how–who communication skill bank: a hierarchy of reusable nodes, each holding an applicability condition, an executable instruction, and an output contract. A complete communication path is one selection from each layer, and an LLM-based router ranks candidates for the next layer conditioned on the selection made in the layer before it. This conditional ranking is what makes the shared bank personalized at use time. Around the bank sit two mechanisms: personalized skill routing, which builds each user's path by traversing the conditional ranked lists, and failure-driven skill evolution, which diagnoses failed decisions by layer and responds with skill refinement, skill generation, or rerouting. The whole design translates the abstract idea that different decision states need different advice into an executable loop that can be trained and then fixed for inference.","core_discovery":"AgentCom's central claim is that UserAgents make more accurate final selections when they are allowed to gather evidence from other users' agents, and that this communication must be assembled from reusable skills rather than run as a generic dialogue. The framework decomposes every communication act along four questions: why the target agent needs external advice, what information should be requested, how the advisors should interact, and who the suitable advisors are. Each skill is a node with an applicability condition, an executable instruction, and an output contract; a shared bank stores these nodes hierarchically, personalized routing picks a path for each user and context, and failure-driven evolution adds or refines skills when final choices miss the ground-truth item. The paper's evidence is a set of Hit@1 comparisons in which adding AgentCom improves every backbone on every dataset, including the strongest baseline MemRec.","pith_inferences":["A testable extension the paper does not run is replacing natural-language advisor messages with compact preference vectors; if the gains survive, the value lies in the evidence content rather than in LLM deliberation, and the cost per user would drop sharply.","Because the protocol always places the ground-truth item in the 20-candidate set, the reported Hit@1 is an upper-bound-style accuracy; a harder deployment test would draw candidate sets without the known answer and measure how much of the gain remains.","The failure-driven evolution loop is stated for recommendation, but it is a generic recipe for any agentic decision process where a final choice can be checked against an outcome; transferring the why–what–how–who skill bank to other domains is a direct corollary.","The who layer currently mixes social links and embedding similarity; varying the advisor source alone would reveal whether the gains come from trusted relationships, from preference overlap, or from both."],"forward_implications":["Adding AgentCom to a non-agentic sequential recommender such as SASRec produces the largest relative gains, so even simple backbones can be upgraded by LLM-mediated advisor evidence without retraining.","Adding AgentCom to already-agentic recommenders such as MemRec still helps, so cross-user communication is complementary to interaction memory and internal feedback rather than a substitute for them.","Removing any single layer of the why–what–how–who structure lowers Hit@1, implying each layer answers a distinct question that generic dialogue would leave unresolved.","Disabling failure-driven evolution lowers performance, so a fixed skill bank is not enough; the framework's improvement depends on updating skills from unsuccessful cases.","More user–advisor interaction rounds help up to a point, while more advisor-only discussion consistently hurts, so the cost-quality trade-off has an identifiable optimum."],"supporting_citations":[{"why":"Supplies the SASRec sequential backbone and the candidate sets used to build every evaluation instance.","marker":"[11]"},{"why":"Supplies the GBSR social-recommendation backbone, the comparison that tests whether explicit advisor communication adds value beyond social graph modeling.","marker":"[35]"},{"why":"Supplies the AFL agentic baseline whose feedback loop is compared against AgentCom's communication skills.","marker":"[3]"},{"why":"Supplies the iAgent agentic baseline, an LLM agent acting as a shield between user and recommender.","marker":"[34]"},{"why":"Supplies the MemRec agentic baseline with interaction memory, the strongest original backbone AgentCom must beat.","marker":"[5]"},{"why":"Provides the LastFM dataset with listening histories and friendship links used for training and evaluation.","marker":"[4]"},{"why":"Provides the Epinions dataset with product interactions and trust links used for training and evaluation.","marker":"[2]"},{"why":"Provides the LibraryThing dataset with book interactions and social connections used for training and evaluation.","marker":"[40]"},{"why":"Provides the classical communication-theory why-what-how-who decomposition that structures the skill bank.","marker":"[13]"},{"why":"Supplies empirical evidence on advice taking that motivates why advisor evidence can improve an agent's judgment.","marker":"[36, 37]"}],"fun_headline_variants":["Why-what-how-who skill bank lifts Hit@1 across backbones","Skill-based advisor chats boost recommender accuracy","Personalized communication skills boost agentic recommender accuracy","AgentCom personalizes advisor advice to raise Hit@1","From generic chats to skilled advisor advice for recommender accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the reported Hit@1 gains are caused by AgentCom's communication skills and not by chance variation in the language model's outputs or by the protocol that always inserts the correct item into the candidate set.","fun_headline_variants_meta":{"raw":{"variants":["Why-what-how-who skill bank lifts Hit@1 across backbones","Skill-based advisor chats boost recommender accuracy","Personalized communication skills boost agentic recommender accuracy","AgentCom personalizes advisor advice to raise Hit@1","From generic chats to skilled advisor advice for recommender accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000866,"raw_usage":{"total_tokens":3773,"prompt_tokens":982,"completion_tokens":2791,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":2710}},"tokens_in":598,"tokens_out":2791,"duration_ms":20054,"temperature":1.0,"reasoning_tokens":2710,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:36:05.784838+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Table 2 protocol with at least ten random seeds and report mean, standard deviation, and confidence intervals; if the AgentCom-over-baseline gaps for SASRec and GBSR overlap zero, or if removing the guaranteed ground-truth item from the 20-candidate set eliminates the advantage, the central claim is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the iAgent agentic baseline, an LLM agent acting as a shield between user and recommender."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MemRec agentic baseline with interaction memory, the strongest original backbone AgentCom must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LibraryThing dataset with book interactions and social connections used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the classical communication-theory why-what-how-who decomposition that structures the skill bank."}],"review_version":1}