{"id":"e6fb4ccc-8866-47da-ac73-72789999b34e","arxiv_id":"2412.09237","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LMAgent simulates 10,000 multimodal LLM agents in e-commerce, claiming human-like purchase behavior, matched co-purchase patterns, and emergent herd behavior.","lead":"This paper builds LMAgent, a virtual society of more than 10,000 AI agents that chat, shop, and watch live-streams inside an e-commerce sandbox. It reports that the agents' purchases and co-purchase patterns resemble real JD.com user data, and that a herd effect emerges as the society grows.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing pretraining control: the co-purchase and herd-behavior results in §IV.E.1 could be produced by GPT-4's memorized e-commerce knowledge instead of LMAgent's social/memory mechanisms.","rationale":"After reading the paper in good faith, the system is genuinely substantial: a 10,000-agent multimodal architecture with memory abstraction and small-world initialization, and the purchase-prediction numbers are suggestive. The reader's conditional verdict is appropriate. However, the flagship external-validity claims, co-purchase resemblance and emergent herding, rest on a single comparison to JD data with no control for the LLM's pretraining. This is a standard confound in LLM-agent evaluation: if a control-free GPT-4 prompt can reproduce the category-level PMI pattern and product concentration, the simulation mechanics add nothing beyond the base model. The proposed zero-shot test would settle this. The reader already flagged this exact assumption and assigned CONDITIONAL, so my pass does not change the verdict.","tokens_in":15620,"tokens_out":4767,"duration_ms":53847,"concrete_test":"Run a zero-shot control using the same gpt-4-1106-preview model and the same eight JD product categories and catalog: prompt it directly, with no agents, no memory, and no social network, to rate pairwise category co-purchase likelihood and to rank the catalog's most popular products. Convert these outputs into a category-level PMI matrix and a top-product concentration curve, then compute rank agreement (e.g., Kendall's tau) against Fig. 5(b) and Fig. 5(c). If the zero-shot prior reproduces the same high-positive pairs (video games/cell-phone accessories), the same high-negative pair (industrial supplies/art crafts), and the same concentration ordering, the claimed causal role of LMAgent's social and memory mechanisms is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a 10,000-agent society produces co-purchase patterns resembling JD data and emergent herd behavior, and that these arise from the simulation's social and memory mechanisms. The experiments never test this causal attribution. The evaluation in §IV.E.1 compares a category-level PMI matrix from 10,000 agents against JD co-purchase data, but there is no control condition in which the same product catalog, personas, shopping pipeline, and recommendation logic are run without chatting, posting, or live-streaming, and no zero-shot elicitation of GPT-4's prior over category co-occurrence. Because the eight product categories are broad and commonplace (video games, cell-phone accessories, industrial supplies, art crafts), GPT-4's pretraining plausibly contains exactly these associations; the same is true of the most-purchased products behind the top-1 concentration curve in Fig. 5(c). In the purchase-prediction task in §IV.B and Table I, the candidate list consists of real Amazon products, so the high accuracy may reflect recognition of popular items rather than simulated user dynamics. The ablations in Table V remove memory, multimodal input, and self-consistency prompting, but they never remove the LLM's parametric prior or the social graph; they therefore cannot separate pretraining effects from emergent social behavior. Without such a control, the 'striking resemblance' and 'herd behavior' claims are not supported by the presented evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LMAgent, a large-scale multimodal LLM-agent society instantiated in an e-commerce sandbox. The system couples agent personas, a fast memory mechanism, a small-world social network, and multimodal shopping and social behaviors with a self-consistency prompting scheme. The experiments evaluate purchase prediction on held-out Amazon purchase histories, human-rated behavior chains and content, social-influence effects, ablations of the proposed components, token-efficiency gains, and large-scale (up to 10,000-agent) simulations whose co-purchase patterns are compared with JD data. The headline claims are that LMAgent achieves state-of-the-art purchase-prediction accuracy, produces behavior close to human benchmarks, saves about 40% of tokens, and, at scale, reproduces real-world co-purchase structure and exhibits emergent herd behavior.","tokens_in":15908,"tokens_out":9053,"duration_ms":90251,"significance":"If the large-scale claims hold, LMAgent would be a notable infrastructure contribution: it demonstrates that multimodal LLM agents can be run at 10,000-agent scale with memory and social interaction, and it provides a concrete external holdout benchmark for purchase prediction with large margins over existing agent-based baselines. The fast-memory token-consumption measurement is concrete, and the paper includes ablations and a human-evaluation protocol with reported inter-annotator agreement. The main weakness is that the causal attribution of the co-purchase resemblance and herd behavior to the simulation's social and memory mechanisms is not established, and the quantitative comparisons lack error bars and significance tests. These issues are addressable with additional control experiments and statistical reporting, so the central contribution remains defensible.","major_comments":[{"comment":"The paper's central large-scale claims—that LMAgent produces co-purchase patterns with 'a striking resemblance' to JD data and exhibits emergent 'herd behavior'—are not supported by the present evidence because no condition removes the LLM's parametric prior or the social/memory machinery. The comparison in Fig. 5(a)-(b) uses eight broad, commonplace categories (e.g., video games, cell-phone accessories, industrial supplies), so GPT-4's pretraining may already encode the same co-occurrence associations; Fig. 5(c) measures only top-product concentration and does not show that agents' choices are causally influenced by other agents. The ablations in Table V remove fast memory, multimodal input, and self-consistency prompting, but they never disable chatting, posting, live-streaming, or the social graph. A concrete control would run the identical shopping pipeline with social interactions disabled and with a zero-shot prompt that elicits GPT-4's category-level PMI prior; the paper should also report a quantitative alignment measure between the simulated and JD PMI matrices rather than visual similarity. The same concern applies to Table I, where the candidate list contains real Amazon products; a control using product metadata alone would clarify how much of the gain is due to product recognition rather than simulated user dynamics.","section":"§IV.E.1, Fig. 5, Table V"},{"comment":"The 'herd effect' is not directly measured. The reported quantity is the share of purchases going to top-ranked products as a function of agent count, and this can increase with scale for reasons unrelated to social influence, such as the recommender system's exposure bias, the popularity distribution of the product catalog, or GPT-4's tendency to choose well-known products. The paper does not report a no-social baseline, product exposure frequencies, or an interdependence metric among agents' purchase decisions. Without these, the scale-dependent concentration in Fig. 5(c) cannot be attributed to emergent collective behavior. The authors should compare the scale curve against independent agents with no social network and against a random-choice population drawn from the same exposure distribution.","section":"§IV.E.1, Fig. 5(c)"},{"comment":"The quantitative performance claims are made without error bars, confidence intervals, or significance tests. The statement in §IV.B.1 that 'due to the large scale of experiments and the independence of agents' actions, it inherently avoids randomness issues' is not a statistical argument; independence across agents does not remove seed sensitivity or sampling variability. Table I reports single numbers for each model, and Table V reports differences of 0.28%, 2.47%, and 4.27% about which the reader cannot judge significance. The claims of 'significantly improved' and 'negligible impact' require repeated runs (at least several seeds) and appropriate tests. Fig. 5(c) also lacks error bars across simulation repeats.","section":"§IV.B.1, Eq. (11), Tables I and V"}],"minor_comments":[{"comment":"The text '10, 100, 1,000, and 10,1000 agents' should read '10, 100, 1,000, and 10,000 agents'; there are also typographical issues such as '˙As shown' and 'A VG' in Table I.","section":"§IV.E.1"},{"comment":"The paper does not describe how products from the Amazon Review Dataset are mapped to the eight JD categories used in the co-purchase analysis; without this mapping, the PMI comparison in Fig. 5(a)-(b) is difficult to interpret.","section":"§IV.E.1"},{"comment":"Algorithm 1 uses the condition 'r < p' for rewiring, while Eq. (9)-(10) use 'r ≤ p'; the two should be made consistent.","section":"§III.D.1"},{"comment":"In Table II, the Random row's human 'Social Norms' score of 4.33 is surprisingly high for agents taking arbitrary actions and should be checked for a typo or explained.","section":"§IV.C.3, Table II"},{"comment":"The name 'self-consistency prompting' may be confused with the standard self-consistency decoding method in LLM literature; the paper should clarify the difference, since the mechanism here is a two-stage chain-of-thought prompt construction.","section":"§III.C.2"},{"comment":"The 40% token-saving figure is measured with a 100-agent society; the paper should state explicitly whether the same efficiency gain is expected or verified at the 10,000-agent scale used in the large-scale experiments.","section":"§IV.D.1, Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The system contribution is plausible and the purchase-prediction benchmark is a real strength, but the headline claims about emergent herd behavior and co-purchase resemblance need additional control conditions to separate simulation dynamics from GPT-4's pretraining. The missing controls and absence of statistical reporting are the main risks, but both are addressable within the manuscript's scope. The paper would also benefit from stating whether code or simulation logs will be released, since the claimed 10,000-agent society is otherwise hard to reproduce independently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine engineering effort with a clean external benchmark, but the paper oversells its emergent-behavior results because it never tests whether GPT-4's prior knowledge, rather than the simulation's social and memory mechanisms, produces the observed patterns.\n\nWhat is new: LMAgent combines known pieces—Generative Agents-style memory, RecAgent-style purchase prediction, small-world topology, multimodal inputs—into a system that scales to 10,000 agents. The fast-memory cache is a sensible engineering contribution, cutting token use by roughly 40% with negligible performance loss. The self-consistency two-stage prompting is plausible and the ablation in Table V shows it helps. The purchase-prediction task uses held-out Amazon purchase histories and beats RecAgent by a large margin; that is a legitimate external benchmark, not a fitted target, and it is the strongest evidence in the paper.\n\nThe soft spots are real. Section IV.E compares simulated co-purchase PMI with JD data, but there is no control condition: no agents without chatting or posting, no zero-shot elicitation of GPT-4's category associations, no random-persona baseline. The eight categories (video games, phone accessories, industrial supplies) are broad and commonplace; GPT-4 almost certainly carries priors for exactly these associations. The same is true for the top-product concentration curve—with larger populations, the model may simply be converging on popular items it already knows, not producing an emergent herd effect. The ablations remove memory, multimodal input, and self-consistency prompting, but never remove the LLM's parametric prior or the social graph, so they cannot separate pretraining from emergent behavior. Without that control, the \"striking resemblance\" and \"herd behavior\" claims are not supported by the evidence.\n\nThere are also statistical issues. No error bars or significance tests anywhere, and the statement that large-scale experiments \"inherently avoid randomness issues\" is not a statistical argument. Table IV is sloppy with percent versus percentage-point changes. Code and data are not released, so reproducibility is limited. On the positive side, the GPT-4 evaluator bias is honestly acknowledged, and the human-evaluation setup, while under-explained, is a reasonable attempt at validating behavioral plausibility.\n\nBottom line: LMAgent is promising, the purchase-prediction result is solid, and the efficiency mechanism is useful. But the central validation of authentic co-purchase and emergent herding needs a proper control condition and better statistics. I would send this to peer review—a serious referee can demand those additions—but I would not take the emergent-behavior claims at face value yet.","headline":"A real scaling contribution with a solid purchase-prediction benchmark, but the flagship herding and co-purchase claims need a pretraining control before they are credible.","tokens_in":16440,"tokens_out":1995,"would_cite":true,"duration_ms":21668,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LMAgent claims a 10,000-agent society of multimodal LLMs reproduces real-world co-purchase structures and emergent herd behavior in an e-commerce sandbox.","keywords":["multi-agent simulation","large language models","multimodal agents","e-commerce","co-purchase patterns","herd behavior","small-world network","memory mechanism"],"falsifier":"Run the same 10,000-agent simulation with the social network removed or with personas replaced by random profiles, then compute the co-purchase PMI matrix and top-product concentration and compare them to the JD baseline. If the match is unchanged or remains equally strong, the claimed simulation dynamics are not the source of the observed alignment.","tokens_in":15421,"feed_emoji":"🛒","tokens_out":3151,"duration_ms":30614,"temperature":0.7,"pith_summary":"This paper sets out to establish that a very large-scale society of multimodal LLM agents can simulate believable multi-user behavior, using e-commerce as a concrete test case. The authors claim that with up to 10,000 agents, each equipped with a persona, memory, and social network, the aggregate purchasing patterns closely resemble real user co-purchase data and even exhibit herd behavior. If true, this would make such agent societies a viable tool for studying complex social and consumer phenomena without recruiting human participants. The paper also introduces efficiency mechanisms to make such large-scale simulations computationally feasible.","feed_headline":"10,000-agent AI society reproduces real shopping patterns","feed_subtitle":"A multimodal, 10,000-agent sandbox matches JD co-purchase data and shows herding at scale.","key_machinery":"Three components carry the argument. Self-consistency prompting splits each decision into two stages, first summarizing the agent's persona and recent observation, then combining that summary with multimodal product information to choose an action, improving decision consistency. Fast memory mechanism caches embeddings and importance scores for routine behaviors in a memory bank, cutting token consumption by roughly 40% and enabling 10,000-agent runs. The small-world network initialization, built by rewiring a ring lattice, gives the society high clustering and short average path lengths, accelerating information spread while resembling real social networks.","core_discovery":"The central claim is that LMAgent, a sandbox society of multimodal LLM agents, produces aggregate consumer behavior that matches real-world patterns. Specifically, co-purchase matrices computed from 10,000-agent simulations align with JD user data in key ways: high intra-category co-purchase frequency, a strong cross-category link between video games and cell-phone accessories, and a negative association between industrial supplies and art crafts. Additionally, as the agent count grows from 10 to 10,000, purchasing becomes increasingly concentrated on top products, with the most-purchased item reaching nearly 30% of all purchases, which the authors interpret as a herd effect mirroring real consumer behavior.","pith_inferences":["The paper's validation lacks a control condition that isolates the simulation's social and memory dynamics from GPT-4's pretraining; a reader should ask whether the LLM already knows the co-purchase regularities found in the evaluation data, which would make the alignment less informative.","A natural extension would be to rerun the 10,000-agent simulation with social interactions disabled or with random personas, to see whether the co-purchase structure and top-product concentration persist; if they do, the herd-behavior and authenticity claims would be substantially weakened.","The herd-behavior interpretation could be sharpened by comparing the scale-dependent concentration curve against a null model of independent agents making noisy choices, which would clarify whether the effect is genuinely emergent or a consequence of recommender feedback and statistical aggregation."],"forward_implications":["If correct, LLM agent societies could serve as large-scale substitutes for human subjects in e-commerce and social-science experiments, capturing aggregate behaviors that small-group studies cannot.","The claimed alignment of co-purchase patterns suggests that simulated markets could be used to test marketing strategies or recommendation algorithms before deployment.","The efficiency gains from fast memory (approximately 40% token reduction with negligible performance loss) lower the cost barrier for running thousand-plus-agent simulations.","Multimodal inputs plus two-stage prompting improve purchase prediction accuracy over text-only agents, indicating that visual product information materially affects simulated consumer decisions.","The architecture is presented as generalizable beyond e-commerce, so the same society framework could be adapted to other multi-user domains such as traffic, finance, or opinion dynamics."],"supporting_citations":[{"why":"Supplies the generative-agent memory, planning, and reflection architecture that LMAgent builds on.","marker":"[2]"},{"why":"The underlying GPT-4 model used as the multimodal LLM engine for all agent decisions.","marker":"[3]"},{"why":"Provides the RecAgent baseline and the a@(a+b) purchase-prediction metric that LMAgent is compared against.","marker":"[6]"},{"why":"Chain-of-thought prompting, which the self-consistency prompting mechanism extends into two stages.","marker":"[33]"},{"why":"The Watts-Strogatz small-world model that initializes the society's social network for realistic information spread.","marker":"[34]"},{"why":"The Amazon Review Dataset supplies product information and real user purchase histories for initializing agents and evaluation.","marker":"[36]"},{"why":"Pointwise mutual information is the metric used to compare simulated co-purchase patterns with JD user data.","marker":"[46]"},{"why":"Provides the background on herd behavior in complex adaptive systems that motivates the interpretation of scale-dependent purchase concentration.","marker":"[47]"}],"fun_headline_variants":["10k AI agents replicate real co-purchase patterns","Multimodal agent society matches JD shopping data with herding","Agent society of 10,000 mirrors real e-commerce behavior","AI shoppers at scale mimic real co-buying and herd effects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The validation assumes GPT-4's pretraining has not already encoded the purchasing regularities found in the Amazon and JD e-commerce data used for evaluation, since the paper offers no control condition (such as agents without social interaction or with random personas) to rule out the model recalling popular products and typical associations.","fun_headline_variants_meta":{"raw":{"variants":["10k AI agents replicate real co-purchase patterns","Multimodal agent society matches JD shopping data with herding","Agent society of 10,000 mirrors real e-commerce behavior","AI shoppers at scale mimic real co-buying and herd effects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1467,"prompt_tokens":909,"completion_tokens":558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":488}},"tokens_in":525,"tokens_out":558,"duration_ms":5374,"temperature":1.0,"reasoning_tokens":488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:11:09.904675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 10,000-agent simulation with the social network removed or with personas replaced by random profiles, then compute the co-purchase PMI matrix and top-product concentration and compare them to the JD baseline. If the match is unchanged or remains equally strong, the claimed simulation dynamics are not the source of the observed alignment.","supporting_citations":[{"cited_title":"Generative agents: Interactive simulacra of human behavior,","cited_arxiv_id":null,"evidence_quote":"Supplies the generative-agent memory, planning, and reflection architecture that LMAgent builds on."},{"cited_title":"GPT-4 technical report. https://openai.com/gpt-4,","cited_arxiv_id":null,"evidence_quote":"The underlying GPT-4 model used as the multimodal LLM engine for all agent decisions."},{"cited_title":"User behavior simulation with large language model-based agents,","cited_arxiv_id":null,"evidence_quote":"Provides the RecAgent baseline and the a@(a+b) purchase-prediction metric that LMAgent is compared against."},{"cited_title":"Amazon review dataset (2018),","cited_arxiv_id":null,"evidence_quote":"The Amazon Review Dataset supplies product information and real user purchase histories for initializing agents and evaluation."},{"cited_title":"Word association norms, mutual information, and lexicography,","cited_arxiv_id":null,"evidence_quote":"Pointwise mutual information is the metric used to compare simulated co-purchase patterns with JD user data."},{"cited_title":"Herd behavior in a complex adaptive system,","cited_arxiv_id":null,"evidence_quote":"Provides the background on herd behavior in complex adaptive systems that motivates the interpretation of scale-dependent purchase concentration."}],"review_version":1}