{"id":"cb8cf0df-606f-4487-a098-851da9de8988","arxiv_id":"2506.12331","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"IndoorWorld is a new multi-agent environment that combines physical task solving with social interaction, and its experiments show effects of collaboration, resource competition, and layout on agent behavior.","lead":"The authors built IndoorWorld, a simulated office where AI agents with different roles, abilities, and knowledge work on physical tasks and talk to each other. It is designed to test AI agents and to help architects predict how people might use real building spaces.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'impact' demonstrations (resource competition, spatial layout) appear to be based on single unreplicated runs, so the headline conclusions in Section 4.3 may be noise rather than signal.","rationale":"The reader's weakest_assumption targets the validity of LLM-generated behavior as a proxy for real office occupants, which is important for the architectural-design application but not for the environment's primary claim as a testbed for LLM multi-agent systems. I see a more immediate, load-bearing weakness in the empirical demonstrations themselves: the resource-competition and spatial-layout experiments appear to be single runs with no variance reporting, while the Limitations section admits substantial run-to-run variability. If those effects vanish under replication, the headline claim that IndoorWorld 'demonstrates the impact' of resource competition and layout collapses, even if the proxy question were answered affirmatively. The task-solving benchmark is somewhat better (three runs are mentioned), but Table 2 still omits variance. I therefore partially agree with the reader: their rationale already notes the replication issue, though their stated weakest_assumption is the proxy validity. My proposed concrete test directly checks whether the central experimental claims are robust. If they are robust, the paper's core testbed contribution stands; if not, the paper needs substantial revision. The reader's CONDITIONAL verdict already captures this need, so my concern does not change the verdict—hence UNCHANGED. I credit the paper for a genuinely integrated environment, a reasonable action-space breadth, and a modular object-oriented design; those parts are not in question. The single most actionable fix is to add replication and statistical reporting to the experiments that substantiate the impact claims, and to release the code as promised.","tokens_in":19616,"tokens_out":4076,"duration_ms":50177,"concrete_test":"Run 10 independent seeds for each condition in Table 3 (e.g., 8 thirsty agents, all-like-water, 1/1 and 2/2 resources) and for each of Design 1 and Design 2 in Section 4.2, using the same models, prompts, and temperature 0.6 as reported. Report mean ± standard deviation (or interquartile range) for hydration time, agent presence percentages, and well-being metrics, and compute bootstrap confidence intervals or a permutation test for the Design 1 vs Design 2 differences. If the inter-run variability is comparable to or larger than the reported effect sizes, the Section 4.3 conclusions do not survive; if the effects are consistent across seeds, the single-run concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 4.3 claim that experiments demonstrate the impact of multi-agent collaboration, resource competition, and spatial layout on agent behavior. For the resource-management experiment, Section 4.2 says 'We ran 3 sets of simulations... and for each setting we ran simulation with different availability'—one simulation per condition, with Table 3 reporting single numbers for agent hydration counts and total time. The spatial-layout experiment similarly says 'We ran simulations in two scenarios... each lasting for an 8-hour simulation period,' with Figures 3–6 showing no error bars or repeated-run bands. The paper's Limitations section explicitly acknowledges 'fluctuations in results' and states only that 'we conducted three independent runs for each task-solving experiment,' not for the resource or layout experiments. Yet Section 4.3 concludes that 'spatial design significantly influences resource accessibility, social interactions, and agent efficiency and well-being' and that the simulation 'can effectively reflect the impact of different resource allocation strategies.' Given stochastic LLM decisions (temperature 0.6, multi-agent feedback loops, acknowledged conversation-driven variability), the observed differences—e.g., Design 2's pantry concentration and reduced well-being, or the 4→6→10 minute hydration times in Table 3—could fall within run-to-run noise. This is a correctness risk for the paper's central experimental claim, independent of the proxy-validity question about real occupant behavior. The environment construction and task-solving ablations are plausible, but the evidence for the 'impact' conclusions is currently underpowered.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents IndoorWorld, a text-based multi-agent environment for LLM-driven agents that integrates physical object manipulation with social interactions, using heterogeneous agents that differ in role, action space, capability, and knowledge. The environment supports task-solving sessions with hierarchical multi-step tasks and simulation sessions without explicit objectives. The authors evaluate three LLMs on a collaborative office-event benchmark with ablations of the task-prioritization and semantic-map/task-progress modules, run resource-competition and spatial-layout simulations, and collect surveys from 20 and 9 architects to support claims about heterogeneity realism and design relevance. The central claims are that IndoorWorld 'seamlessly integrates' physical task solving and social simulation and that it is a promising tool for LLM-based building occupant simulation for architectural design.","tokens_in":19790,"tokens_out":5152,"duration_ms":63437,"significance":"If the results hold, IndoorWorld is a genuinely useful testbed: it is one of the few environments that couples grounded object-state changes with free-form multi-agent dialogue, and the explicit multi-level heterogeneity (profile, action space, capability, knowledge) goes beyond the homogeneous action spaces common in task-solving benchmarks. The task-prioritization module is a simple, plausible mechanism for improving LLM task focus, and the ablation across three models suggests it helps, especially for open-source models. The architect surveys are a welcome attempt to ground design choices in professional practice. I found no circular reasoning: no parameters are fitted, and the task-prioritization module is an engineered addition evaluated by ablation. However, the central experimental demonstrations of resource competition and spatial layout currently rest on unreplicated simulations, and the task-solving results are reported without variance, so the strength of the current evidence is lower than the conclusions in Section 4.3 suggest.","major_comments":[{"comment":"The resource-management and spatial-layout experiments report a single simulation per condition. Section 4.2 describes running 'simulation with different availability' without mentioning multiple seeds, Table 3 lists single scalar values for each cell, and Figures 3–6 show no error bars or per-run bands. The Limitations section acknowledges that LLM results fluctuate and explicitly states that three independent runs were conducted only 'for each task-solving experiment.' With temperature 0.6 and coupled multi-agent decisions, the observed differences (e.g., Design 2's pantry concentration or the hydration times in Table 3) could reflect run-to-run noise. The conclusions in Section 4.3 that 'spatial design significantly influences resource accessibility, social interactions, and agent efficiency and well-being' and that the simulation 'can effectively reflect the impact of different resource allocation strategies' therefore outrun the evidence. The authors should rerun each resource and layout condition multiple times and report distributions, per-run values, or error bars, and where possible a statistical comparison.","section":"§4.2, §4.3, Table 3, Figures 3–6"},{"comment":"Table 2 reports a single instance-level and attribute-level completion value per model and ablation condition, yet the Limitations state that three independent runs were conducted for every task-solving experiment. No variance, standard error, confidence intervals, or per-run values are given, so the reader cannot assess whether the reported differences (e.g., GPT-4o Full Model IS 79.3 vs. Full Model w/o TP 74.7, or Llama 3.3 Full Model IS 55.2 vs. 31.0) are larger than run-to-run variability. The qualitative examples in Section 4.3 are illustrative but not a substitute. Please report per-run results and dispersion measures, and supplement the qualitative examples with quantitative evidence that the task-prioritization gains are stable across runs.","section":"§4.1, Table 2, Limitations"},{"comment":"The architectural-design application rests on the assumption that LLM-generated agent behavior is a valid proxy for real office occupant behavior. The only supporting evidence is the architect surveys, and Table 6 shows that the surveyed architects are neutral on exactly this point: q5 (actionable insights from resource-competition simulation) has a mean of 3.1 and q8 (agent behavior reflects realistic office usage patterns) has a mean of 3.0. The paper should either temper the conclusion that IndoorWorld is a 'promising tool for architectural design' or add a validation component, such as a qualitative or quantitative comparison with real occupancy data, established occupant-behavior models, or prior occupant-simulation literature. Without such grounding, the architectural claim is an aspiration rather than a demonstrated result.","section":"§1, §4.3, Table 6"}],"minor_comments":[{"comment":"The labels in Figure 2 are garbled (for example 'Design Showcase are Open area' and 'Pantr'); please regenerate the figure with clean labels.","section":"Figure 2"},{"comment":"The header 'A VG' and the inconsistent row spacing make the table difficult to read; please use clear column labels for the five tasks, Average IS, and Average AS.","section":"Table 2"},{"comment":"The caption should clearly explain the triple notation (water drinkers / coffee drinkers / total hydration time) and what '1/1' and '2/2' mean; the current sentence about the 'second row' is confusing.","section":"Table 3"},{"comment":"Several references are incomplete: 'Wang et al.' in the introduction and 'Miller' in the Limitations lack publication years, and some entries do not include full bibliographic details.","section":"References"},{"comment":"The appendix notes that the code examples are simplified and may not match the actual source. Since code is only promised after acceptance, please consider releasing a minimal runnable example or anonymized repository with the paper to support reproducibility.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable contribution for a venue that accepts environment/testbed papers, but the experimental rigor around the 'impact' demonstrations needs substantial improvement. The stress-test concern about unreplicated resource and layout simulations is real and independently confirmed by reading Section 4.2 and the Limitations. I would ask the editor to require revised experiments with multiple seeds and proper dispersion reporting, not just text changes. The neutrality of architect ratings on realism (q8 = 3.0) should also be reflected in the authors' claims. The environment itself is valuable; the revision should focus on matching the strength of the conclusions to the strength of the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real environment paper, not a hype piece. The authors built a heterogeneous multi-agent office simulator where agents differ in role, action space, capability, and knowledge, and where conversations can change what agents do next in a physically grounded world. The task-solving benchmark—five coordinated subtasks, six agents, decentralized division of labor—is a reasonable stress test, and the task-prioritization ablation is informative. Removing the module degrades all three models, with the open-source ones hit hardest. That is a clean, useful result. The two architect surveys are also a plus, and I give the authors credit for reporting the low scores honestly rather than burying them.\n\nThe soft spots are real and, for the central 'impact' claims, load-bearing. The resource-management experiment runs one simulation per condition; Table 3 reports single numbers for hydration counts and times. The spatial-layout experiment compares two 8-hour runs, one per design, with no error bars in Figures 3–6. Given temperature 0.6, multi-agent feedback loops, and the paper's own Limitations section admitting run-to-run fluctuations, the differences the authors highlight—Design 2's pantry concentration, 4 vs 6 vs 10 minute hydration times—could easily be noise. The stress-test note is correct on this. The fix is not philosophical; it is standard practice: repeat the runs, report means and variance, and test whether the contrasts survive. At the stated scale with GPT-4o-mini, that is completely doable.\n\nThe architectural-design claim also overreaches. Section 4.3 says the simulation 'can effectively reflect the impact of different resource allocation strategies,' but the only external evidence is small architect surveys, not comparison with real occupancy data. The architects themselves rated realism at 3.0/5 on q8 and actionable resource insights at 3.1/5. If the paper claims IndoorWorld is a demonstrative platform for exploring hypotheses, fine. The abstract should not imply predictive validity for real buildings.\n\nNo circularity burden here: the task-prioritization module is an engineered addition evaluated by ablation, and the surveys are external human judgments. The literature comparison is fair and appropriately cites Smallville, MineLand, AdaSociety, and others.\n\nWho this is for: LLM multi-agent researchers who want a unified testbed with heterogeneous roles and physical grounding, and secondarily people building LLM-based occupant simulators. The environment design and task-solving ablation deserve a serious referee. I would send it to review, but with a clear expectation of major revision: replicate the resource and layout experiments, report variance, tighten the architectural language, and release code at camera-ready.","headline":"Genuine environment contribution with a solid task-solving ablation, but the resource/layout claims rest on single unreplicated runs and need major revision before the architectural conclusions can be taken seriously.","tokens_in":20387,"tokens_out":2071,"would_cite":false,"duration_ms":27393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IndoorWorld claims that physical task solving and social simulation can be integrated in one world-state engine, and demonstrates the integration through office-task collaboration, resource competition, and spatial-layout experiments.","keywords":["multi-agent systems","LLM agents","social simulation","task solving","heterogeneous agents","building occupant simulation","spatial layout","resource competition"],"falsifier":"Run the two spatial-layout scenarios and compare the simulated agents' location-presence, movement, and unmet-need time series against measured activity data from two comparable real office layouts. If the layout that looks worse in simulation does not also shift real occupants' presence and well-being in the predicted direction, the occupant-simulation claim would be falsified even though the environment might still work as an LLM-agent benchmark.","tokens_in":19358,"feed_emoji":"🏢","tokens_out":9371,"duration_ms":99248,"temperature":0.7,"pith_summary":"This paper introduces IndoorWorld, a multi-agent environment in which LLM-driven agents must solve physical office tasks while managing social interactions. The central claim is that the two can be tightly coupled: conversations change task progress and knowledge, while physical locations, objects, and state variables give social behavior a real referent. This coupling, the paper argues, supports agents that differ in role, action, capability, and knowledge, and it produces behavior that purely physical or purely social benchmarks miss. Experiments with an event-preparation benchmark, resource competition, and two office layouts show the environment responding to collaboration, scarcity, and spatial arrangement, setting up IndoorWorld as a candidate building-occupant simulator for architectural design.","feed_headline":"LLM agents that chat while working reveal how offices shape behavior","feed_subtitle":"By anchoring talk to objects and locations, the simulator makes layout and resource choices visible in agent behavior.","key_machinery":"The load-bearing mechanism is an object-oriented world state, with agents, objects, and locations as stateful entities whose actions have preconditions and effects, wrapped in a five-module agent architecture (perception, memory, planning, action, task prioritization). Heterogeneity enters at four levels—profile, action space, capability, and knowledge—so the same world state yields different admissible actions and efficiencies across agents. Conversations are treated as actions anchored to a shared location, which is what physically grounds social behavior. The task-prioritization module is the third piece: it re-reminds agents of incomplete tasks and objects, counteracting LLM agents' tendency to switch tasks mid-way, and the paper shows that removing it degrades completion rates.","core_discovery":"On the paper's own terms, the discovery is that a world-state engine can hold task solving and social simulation together without letting either collapse into the other. Every object, location, and agent carries explicit state variables; admissible actions are gated by role and capability; conversation is itself a location-bound action that updates internal state and task progress. The result is that agents negotiate labor division, request help for tasks beyond their strength, share passwords to book rooms, and compete for limited water and coffee, while their physiological needs decay and pull them toward resources. The paper interprets these dynamics as evidence that IndoorWorld is a testbed for heterogeneous LLM agents and a potential tool for studying occupant behavior in architectural design.","pith_inferences":["A natural test is to compare IndoorWorld's spatial-layout predictions against sensor or diary data from a real office; if simulated presence and movement match human traces, the architectural-design use case is substantially strengthened.","The role-gated action space is a generic pattern, so the same JSON-defined engine could be adapted to hospitals, labs, or classrooms where access rights and capacity constraints shape behavior.","The task-prioritization results suggest a testable hypothesis: weaker open-weights planners benefit most from external memory scaffolding, implying that structured progress reminders may be a broadly useful design for LLM multi-agent systems."],"forward_implications":["The five-task event-preparation benchmark provides a reproducible setting for decentralized labor division, task prioritization, and coordination among heterogeneous LLM agents.","Resource competition experiments show that adding water dispensers or coffee machines shortens the time until all agents are hydrated, while adding agents lengthens it.","Spatial-layout experiments show that moving the same functional areas changes where agents spend time, how much they move, and how often their needs go unmet; Design 1 yielded more balanced area occupancy than Design 2.","Removing the task-prioritization module lowered task completion rates across Llama 3.3, Gemma 3, and GPT-4o, with the largest drops for the open-weights models.","Architect surveys rated multi-level heterogeneity as more realistic and important, with spatial-layout simulation rated especially helpful for understanding occupant behavior."],"supporting_citations":[{"why":"Establishes the generative-agent social-simulation approach that IndoorWorld contrasts by adding physical grounding to conversations.","marker":"Park et al., 2023"},{"why":"Provides the ALFWorld physical task-solving baseline, where agents interact with objects but lack social heterogeneity.","marker":"Shridhar et al., 2021"},{"why":"Provides the VirtualHome embodied-task baseline with homogeneous agents, motivating role and capability heterogeneity.","marker":"Puig et al., 2018"},{"why":"Supplies the ReAct reasoning-and-acting loop that IndoorWorld's action module builds on.","marker":"Yao et al., 2022"},{"why":"Supplies the COELA memory design, including the semantic map and task-progress tracking that IndoorWorld's memory module adapts.","marker":"Zhang et al., 2024"},{"why":"Frames occupant behavior modeling as a building-performance problem that IndoorWorld targets for architectural design.","marker":"Yan et al., 2015"},{"why":"Provides the simulation-of-occupancy background underpinning the proposed building-occupant simulation application.","marker":"Feng et al., 2015"}],"fun_headline_variants":["Social chats meet physical tasks in multi-agent offices","Agents that talk and toil: IndoorWorld blends both","Virtual offices where agents chat and shape space","LLM agents balance talk and tasks in shared spaces","IndoorWorld: where agent gossip affects the layout"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that LLM-driven behavior in this text-based office is a valid proxy for how real office occupants use space, so layout and resource differences in the simulation predict human behavior.","fun_headline_variants_meta":{"raw":{"variants":["Social chats meet physical tasks in multi-agent offices","Agents that talk and toil: IndoorWorld blends both","Virtual offices where agents chat and shape space","LLM agents balance talk and tasks in shared spaces","IndoorWorld: where agent gossip affects the layout"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000152,"raw_usage":{"total_tokens":1129,"prompt_tokens":797,"completion_tokens":332,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":413,"completion_tokens_details":{"reasoning_tokens":257}},"tokens_in":413,"tokens_out":332,"duration_ms":4504,"temperature":1.0,"reasoning_tokens":257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:52:11.364314+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the two spatial-layout scenarios and compare the simulated agents' location-presence, movement, and unmet-need time series against measured activity data from two comparable real office layouts. If the layout that looks worse in simulation does not also shift real occupants' presence and well-being in the predicted direction, the occupant-simulation claim would be falsified even though the environment might still work as an LLM-agent benchmark.","supporting_citations":[{"cited_title":"O'Brien, Carrie J","cited_arxiv_id":null,"evidence_quote":"Establishes the generative-agent social-simulation approach that IndoorWorld contrasts by adding physical grounding to conversations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VirtualHome embodied-task baseline with homogeneous agents, motivating role and capability heterogeneity."}],"review_version":1}