{"id":"b693c722-9407-4b04-b51c-5673607c3063","arxiv_id":"2412.10428","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using Schelling's model, the authors find that LLM-generated move/stay advice leads to a similarly segregated society across models and demographic categories, regardless of the model's measured bias.","lead":"This paper uses Thomas Schelling's segregation model to simulate what happens when people follow large language model (LLM) advice about where to live. It reports that even LLMs that score well on bias tests still push society toward segregation once enough people follow their suggestions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline segregation result is manufactured by the prompt's explicit Rule 2, which instructs LLMs to prefer same-type neighbors; without a neutral-prompt control the central claim is unsupported.","rationale":"The reader's weakest-assumption analysis correctly identifies the prompt's Rule 2 as the load-bearing flaw. The abstract claims segregation emerges 'regardless of the level of bias in LLMs,' but the only evidence is a Schelling simulation in which the LLM is explicitly told to be more satisfied with same-type neighbors. This is a textbook case of the conclusion being embedded in the input. The paper also lacks a random baseline and an explicit rating-to-threshold mapping, but those are secondary; even if the mapping were principled, the explicit preference instruction would still invalidate the claim that the observed micromotive is intrinsic to the LLM. The proposed concrete test—running an inverted or neutral prompt—would settle the matter directly. Since the reader already recommends REJECT and our analysis supports that, no verdict change is needed.","tokens_in":13211,"tokens_out":2267,"duration_ms":24692,"concrete_test":"Rerun the full pipeline with two control conditions: (1) an inverted prompt replacing Rule 2 with 'satisfaction is higher if many individuals of the opposite type are nearby,' and (2) a neutral prompt that omits Rule 2 entirely and simply asks the LLM whether it would move or stay, deriving the threshold from the LLM's natural response. Keep all other grid, iteration, and rating parameters identical. If SegShift remains approximately 0.27 under the inverted rule, the effect reflects intrinsic LLM behavior; if it flips sign or drops to near zero, the headline result is an artifact of the explicit preference instruction. Also run a random-threshold baseline to establish the null level of segregation under the same movement mechanics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM micromotives, even when bias is low, drive a highly segregated society. But the operationalization builds that conclusion into the prompt: Appendix A, Rule 2, tells the model 'Satisfaction is higher if many individuals of the same {agent_type} are nearby.' The LLM is then asked to rate 'move' and 'stay' responses on a satisfaction scale, and those ratings are converted into Schelling thresholds. This is not an observation of an intrinsic LLM preference; it is a direct instruction to exhibit exactly the preference that produces segregation. The paper never tests a prompt without Rule 2, nor a prompt with the preference inverted, so it cannot distinguish a property of LLMs from a property of the experimental protocol. A second, compounding gap is that the mapping from satisfaction ratings to movement thresholds is never specified; if the threshold is set near a Schelling tipping point, even noisy ratings would yield a SegShift around 0.27. The paper's own limitation section acknowledges simplifications of real human behavior but does not address this prompt-level confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends Schelling's segregation model by using LLM satisfaction ratings as the movement rule. For five demographic categories and five LLMs, the authors prompt the model to rate \"move\" and \"stay\" responses, use the average ratings as a tolerance threshold, and simulate a 20x20 Schelling grid. They report a consistent segregation shift of approximately 0.27 across all models and categories, and a tipping point beyond 40% LLM adoption. The abstract claims that, regardless of the LLM's bias score, a highly segregated society emerges as more people follow LLM suggestions.","tokens_in":13431,"tokens_out":5451,"duration_ms":48756,"significance":"The question of whether LLM suggestions could shape societal macrobehavior is timely, and the idea of embedding LLMs in a Schelling-style simulation is creative. If the central claim were supported, the paper would connect LLM bias mitigation debates to systemic social outcomes. However, the current operationalization explicitly instructs the LLM to prefer same-type neighbors, so the measured segregation is an engineering consequence of the prompt rather than an observed property of LLM micromotives. The cross-model, cross-category consistency reported in Table 1 is descriptively interesting, but it cannot be interpreted as evidence about intrinsic LLM biases without a neutral-prompt control. The paper also provides no code or data release, limiting reproducibility.","major_comments":[{"comment":"The prompt explicitly states: \"Satisfaction is higher if many individuals of the same {agent_type} are nearby; conversely, it is lower if many individuals of {opp_agent_type} are present.\" This rule directly installs the in-group preference that is known to drive Schelling segregation. The abstract's central claim that \"regardless of the level of bias in LLMs, a highly segregated society will emerge\" is therefore a consequence of the prompt design, not an observed property of LLM micromotives. The manuscript never tests a prompt without Rule 2 or with an inverted rule, so it cannot distinguish an intrinsic LLM preference from an experimenter-imposed one. This confound invalidates the answer to RQ1 and the paper's headline conclusion.","section":"Appendix A (Table 3), Rule 2"},{"comment":"The text states that average satisfaction scores \"serve as moving thresholds in the Schelling model,\" but the exact mapping from the two ratings (yes_rating and no_rating) to a single tolerance threshold t in [0,1] is never specified. Without this formula, the simulation is not reproducible, and the reported SegShift values cannot be checked for sensitivity to the choice of conversion. The authors should provide the exact functional form and a sensitivity analysis over reasonable monotonic transformations of the ratings.","section":"Section 3.3"},{"comment":"The 40% tipping-point claim rests on a partial-adherence simulation whose design is not described: how are the LLM-following agents selected, whether they are re-selected each iteration, and whether \"independent decisions\" are simply random moves with no preference. No error bars or multiple-seed statistics are reported for Figure 1, so the \"critical juncture\" narrative is not supported. In addition, the bias-independence claim compares only two models (GPT-4o and GPT-3.5) in Table 2; with n=2, no statistical conclusion about independence from bias scores is possible, especially given the range of SegShift values for other models (e.g., Gemini's 0.19 to 0.36 in Table 1).","section":"Section 4 (Figure 1 and Table 2)"}],"minor_comments":[{"comment":"In the research questions list, \"(RQ3)How\" should be \"(RQ3) How\".","section":"Section 1"},{"comment":"The phrase \"to accommodation for random initialisation\" should be \"to accommodate for random initialisation\".","section":"Section 3.2"},{"comment":"The column header \"SegShif\" is a typo and should read \"SegShift\".","section":"Table 1"},{"comment":"The initial grid is said to have Seginit approximately 0.511 and to be \"standardized,\" but the exact initial configuration (or the algorithm used to generate it) is not described, which prevents replication.","section":"Section 3.3"},{"comment":"The manuscript does not mention whether code or data will be made available; for a simulation study, a public release would substantially aid reproducibility and trust in the results.","section":"General"}],"recommendation":"reject","confidential_remarks":"The topic is well suited to physics.soc-ph and the experimental idea is original, but the central result is an artifact of the prompt design: Rule 2 in Appendix A explicitly instructs the LLM to prefer same-type neighbors, so the observed segregation is built in. The missing threshold mapping and the weak two-model comparison for the bias-independence claim further undermine the conclusions. A revision with neutral and inverted prompt controls, a precise threshold formula with sensitivity analysis, and a proper statistical treatment could address these issues, but the current manuscript's central claim would require re-testing and would likely change. I therefore recommend reject rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper applies Schelling's segregation model to LLM-generated move/stay decisions and reports that five models across five demographic categories all produce a SegShift around 0.27, with a tipping point when 40% of the population follows LLM advice. The specific setup is new, and the consistency across models is a striking empirical pattern. If the result were genuinely about how LLMs behave outside the prompt, it would be an important challenge to the assumption that removing individual bias is sufficient.\n\nHere is the problem: the prompt tells the LLM exactly what to prefer. Appendix A, Rule 2 states that \"Satisfaction is higher if many individuals of the same {agent_type} are nearby.\" The LLM is asked to rate move/stay responses on that basis. So the micromotive is not observed; it is manufactured. Any instruction-following model will exhibit the same-type preference, and Schelling's model guarantees segregation from that preference. The paper never runs a neutral prompt, an inverted prompt, or a random-threshold baseline. That makes the central claim unsupported as stated.\n\nTwo further issues. The conversion from satisfaction ratings to movement thresholds is never specified. Without that, we cannot tell whether the ratings are doing any work or whether the threshold sits near a Schelling tipping point, where noisy thresholds would produce similar SegShift values regardless of model. Also, no error bars or variance across the ten trials are reported, so we cannot assess whether the 0.27 consistency is meaningful or tight by construction.\n\nThe paper's limitation section acknowledges simplifications of human behavior but does not address the prompt-level confound. That is a notable omission.\n\nWhat is genuinely valuable is the idea of using LLMs to parameterize agent-based models, and the 40% threshold simulation is a nice demonstration of how adoption rates can matter. But the paper needs at least a control condition without the explicit same-type preference before making claims about LLM micromotives. As it stands, it tells us more about prompt compliance than about intrinsic LLM bias.\n\nI would not cite this for the claim, but I might bring it to a reading group to discuss prompt confounds in LLM-driven simulation. It deserves a serious referee because the idea is worth engaging with, but the current version does not support its headline. My recommendation: send it to review, expect heavy revision, and require a neutral-prompt baseline before acceptance.","headline":"Clever application of Schelling to LLM-driven decisions, but the prompt installs the very same-type preference the paper claims to observe, so the central result is unsupported.","tokens_in":13946,"tokens_out":2032,"would_cite":false,"duration_ms":20480,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM advice segregates society even when bias scores are low.","keywords":["large language models","Schelling model","segregation","macrobehavior","micromotives","bias mitigation","agent-based simulation","societal impact"],"falsifier":"Run the identical Schelling experiment with a neutral prompt that describes the neighborhood composition without stating any satisfaction rule, and ask the LLM only to choose move or stay; if the segregation shift drops to near zero, the claim that 'regardless of bias, a highly segregated society will emerge' fails for that configuration.","tokens_in":13016,"feed_emoji":"🏙️","tokens_out":3698,"duration_ms":34421,"temperature":0.7,"pith_summary":"This paper argues that large language models, when asked to advise agents on whether to move based on neighbors' demographics, produce a society that becomes highly segregated—no matter which model is used, which demographic category is tested, or how well the model scores on bias benchmarks. The authors use Schelling's checkerboard model as an observation instrument and report a segregation shift of about 0.27 across all tested configurations. They further show that segregation rises sharply once more than 40 percent of people follow LLM suggestions. The point is that fixing individual-level bias in an LLM may not prevent undesirable society-level outcomes.","feed_headline":"LLM advice segregates society past 40% adoption","feed_subtitle":"Five LLMs and five demographics converge on the same segregated outcome, so debiasing alone may not protect society.","key_machinery":"The central mechanism is the Schelling checkerboard model with an LLM replacing the fixed tolerance threshold. In the experiment, the LLM is asked to rate 'move' and 'stay' responses on a satisfaction scale given the count of same-type and other-type neighbors; these ratings become the movement thresholds. The segregation shift metric compares the initial share of same-type neighbor edges to the final share, normalized by the theoretical maximum similarity. This setup turns the LLM's demographic preferences into a measurable micromotive and reveals the aggregate pattern.","core_discovery":"The paper establishes that the micromotives of LLMs—operationalized as their satisfaction ratings for moving versus staying—lead to a consistent macrobehavior of segregation in a Schelling checkerboard. Five models (GPT-3.5, GPT-4o, Gemini-1.5, Claude-3.5, and Qwen2-72B) across five demographic categories (age, gender, politics, race, and religion) produce segregation shifts around 0.27 even though their bias-benchmark scores differ substantially. The segregation outcome is largely insensitive to the measured bias level, so the common assumption that removing bias from LLMs will make their societal influence benign needs reexamination. The paper also identifies a tipping point: when 40 percent of people use LLM suggestions, segregation starts to increase noticeably, and with full adoption it becomes highly segregated.","pith_inferences":["The prompt's explicit rule that same-type neighbors raise satisfaction may itself be the cause of the observed behavior; testing a neutral prompt that does not state this rule would clarify whether LLMs naturally hold such a micromotive.","If users selectively follow LLM advice only when it matches their own biases, the tipping point could occur at a lower or higher adoption rate than 40 percent, which assumes uniform compliance.","Dynamic trust in LLMs, where users learn to discount advice over time, could dampen or reverse the segregation trend in longer-horizon simulations.","The same framework could be applied to other collective-action problems, such as opinion polarization or resource allocation, by changing the satisfaction rule."],"forward_implications":["Even a model that scores near-perfect on bias benchmarks can still drive segregation when its advice steers individual moves.","Macro-level evaluation of LLM influence should accompany micro-level bias benchmarks, since the two can diverge.","As LLM adoption passes roughly 40 percent, societal segregation grows with each additional user following model advice.","Society-level outcomes, not just model-level debiasing, should be a target for responsible AI deployment."],"supporting_citations":[{"why":"Supplies the segregation model and the micromotives/macrobehavior framework that the entire experiment is built on.","marker":"Schelling (1969)"},{"why":"Provides the LangBiTe bias benchmark scores and the five demographic categories used in the experiments.","marker":"Morales et al. (2024)"},{"why":"Highlights variability in LLM output confidence, used as the rationale for averaging segregation scores over multiple iterations.","marker":"Xu et al. (2024)"},{"why":"Documents GPT-4o, the model used for the adoption-threshold experiment that identifies the 40 percent tipping point.","marker":"OpenAI et al. (2024)"}],"fun_headline_variants":["Even unbiased LLMs still segregate society when followed","LLM suggestions drive segregation past 40% usage","Bias removal won't stop LLM-driven segregation","Schelling's model shows LLMs segregate societies at scale","LLM advice: a segregation tipping point at 40% adoption"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiment's prompt explicitly tells the LLM that satisfaction is higher when more same-type neighbors are nearby; if this rule is not the model's natural preference but an instruction the authors implanted, then the observed segregation is a product of the prompt, not an intrinsic property of the LLMs.","fun_headline_variants_meta":{"raw":{"variants":["Even unbiased LLMs still segregate society when followed","LLM suggestions drive segregation past 40% usage","Bias removal won't stop LLM-driven segregation","Schelling's model shows LLMs segregate societies at scale","LLM advice: a segregation tipping point at 40% adoption"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1646,"prompt_tokens":941,"completion_tokens":705,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":623}},"tokens_in":557,"tokens_out":705,"duration_ms":6521,"temperature":1.0,"reasoning_tokens":623,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:20:33.366808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical Schelling experiment with a neutral prompt that describes the neighborhood composition without stating any satisfaction rule, and ask the LLM only to choose move or stay; if the segregation shift drops to near zero, the claim that 'regardless of bias, a highly segregated society will emerge' fails for that configuration.","supporting_citations":[{"cited_title":"LangBiTe: A Platform for Testing Bias in Large Language Models","cited_arxiv_id":"2404.18558","evidence_quote":"Provides the LangBiTe bias benchmark scores and the five demographic categories used in the experiments."}],"review_version":1}