{"id":"6c66c190-02cb-4022-ba4e-7c53ac4b4b88","arxiv_id":"2509.03956","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"WorMI adapts an LLM-based embodied policy to unseen domains by retrieving and compositionally implanting domain-specific world models at test time.","lead":"A framework called WorMI lets a robot policy made of a fixed large language model and small trainable world models swap those world models in and out at test time, choosing the relevant ones by similarity and fusing them with attention. The authors report large gains over several baselines on VirtualHome and ALFWorld, in both zero-shot and few-shot settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The k-center bound in Eq. (4) is mathematically false: W(E_j,p_j) ≤ ρ does not hold for uniform empirical measures, so the prototype-retrieval justification is unsupported.","rationale":"The reader's verdict of CONDITIONAL and weakest_assumption identify the same load-bearing concern: the proof of the prototype-retrieval bound relies on an invalid inference from the k-center condition to a Wasserstein bound for uniform empirical measures. I confirmed the mathematical gap and constructed a concrete counterexample showing the bound is false in general. This is an internal inconsistency in the paper's theoretical contribution, not just an overclaim. However, the paper's central empirical claim—that WorMI outperforms baselines—is supported by the experiments with confidence intervals and by ablations showing prototype retrieval approximates full retrieval well (Table A.11). The flaw does not directly invalidate the measured success rates, but it does weaken the paper's stated justification for the retrieval method. Since the reader already conditioned acceptance on addressing exactly this and reproducibility issues, my analysis does not change the verdict. If the authors can either correct the bound (e.g., by adding a cluster-imbalance term) or empirically show W(E_j,p_j) is small on their data, the concern would be resolved. No additional fatal issues were identified; the empirical design and baseline comparisons appear reasonable, and the paper honestly reports limitations. I therefore recommend keeping the verdict UNCHANGED rather than moving to accept or reject.","tokens_in":19484,"tokens_out":8032,"duration_ms":75139,"concrete_test":"Verify the bound directly: construct the counterexample E = {0 repeated 99 times, 100 repeated 1 time}, k=2, p = {0,100}, and compute W_1(E,p). If W_1(E,p) = 49 while ρ = 0, the claimed inequality δ(p_i,p_j) ≤ δ(E_i,E_j) + 2ρ is violated. Additionally, on the actual VirtualHome/ALFWorld embedding sets used in the paper, compute W_1(E_j,p_j) for each dataset and compare to the stated ρ; if W_1(E_j,p_j) > ρ for any j, the premise of Eq. (4) is false. This would settle whether the bound fails not only in synthetic cases but also in the paper's real data distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's theoretical justification for prototype-based retrieval rests on the inequality δ(p_i,p_j) ≤ δ(E_i,E_j) + 2ρ (Eq. 4, Appendix A.1). This is derived from the triangle inequality as δ(p_i,p_j) ≤ δ(E_i,E_j) + W(p_i,E_i) + W(p_j,E_j), where p_j are k-center prototypes for the empirical set E_j. The claim that W(E_j,p_j) ≤ ρ is asserted to follow from the k-center condition of Eq. (3): every point in E_j is within ρ of some center. However, W(E_j,p_j) is the Wasserstein distance between the uniform empirical measure on E_j and the uniform measure on the k centers (each with mass 1/k). The k-center condition only guarantees a transport from E_j to the centers with mass assigned to each center proportional to its cluster size, not uniform mass. For imbalanced clusters, the Wasserstein distance can be arbitrarily large even when ρ=0. Concretely, if E_j contains 99 copies of 0 and 1 copy of 100, k=2, the optimal centers are {0,100}, ρ=0, but W_1(E_j,p_j)=49 because uniform center measures require moving 0.49 mass from 0 to 100. Thus Eq. (4) is not a valid bound. This is an internal inconsistency in the paper's technical argument, not just a matter of looseness. The empirical ablations (Table A.11) suggest prototype retrieval performs nearly as well as full retrieval (66.12 vs 66.54 SR), so the central performance claim may still hold in practice, but the theoretical justification in Section 3.2 is unsupported as written. The paper's claim that prototypes 'serve as a bounded proxy' is therefore not established, which is a correctness risk in the framework's rationale.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes WorMI, a framework for test-time adaptation of embodied agents. Domain-specific world models are first pre-trained on datasets from seen domains; at test time, a prototype-based retrieval step selects K relevant world models using Wasserstein distances between cluster prototypes of object-wise state embeddings. A 'world-wise compound attention' module then fuses the intermediate representations of the retrieved world models and aligns them with a frozen LLM-based reasoning model, with the composition module trained by REPTILE-style meta-learning. The method is evaluated on VirtualHome and ALFWorld in zero-shot and few-shot settings, reporting large improvements in success rate and pending steps over ZSP, LLM+FT, LLM-Planner, and SayCanPay, together with ablations, scalability studies, and attention-map analyses.","tokens_in":19986,"tokens_out":5689,"duration_ms":60357,"significance":"If the empirical findings hold, WorMI is a useful step toward modular, test-time composition of domain knowledge in embodied LLM agents. The paper reports 95% confidence intervals, includes ablations showing that both prototype-based retrieval and compound attention contribute to performance, and provides resource-usage comparisons. The gains over SayCanPay are substantial and consistent across two benchmarks and several analysis settings. The main weakness is that the paper's one formal justification for the prototype-based retrieval — the bound in Eq. (4) and Appendix A.1 — is mathematically unsound as written. Since the empirical ablations (Table A.11) show that prototype retrieval performs nearly as well as full retrieval, the central performance claim may still survive, but the theoretical claim needs correction or removal. Some implementation details that are central to reproducing the method are also missing.","major_comments":[{"comment":"The claimed bound δ(p_i,p_j) ≤ δ(E_i,E_j) + 2ρ does not follow from the k-center condition. The proof assumes W(E_j,p_j) ≤ ρ, where p_j is the uniform measure on the k centers. But k-center clustering only guarantees that every point of E_j is within ρ of some center; it does not guarantee that the uniform measure on the centers is close in Wasserstein distance to the uniform empirical measure on E_j. For example, if E_j contains 99 copies of 0 and 1 copy of 100, with k=2, optimal centers are {0,100}, ρ=0, but W_1(E_j,p_j)=49 because the uniform center measure forces 0.49 mass to move from 0 to 100. Thus Eq. (4) is not a valid proxy bound. This is a load-bearing point because it is the stated theoretical justification for prototype-based retrieval. The issue is fixable: define prototypes as cluster-weighted measures (mass proportional to cluster size), in which case W(E_j,p_j) ≤ ρ follow","section":"Section 3.2 / Eq. (4) / Appendix A.1"},{"comment":"The retrieval pipeline is under-specified for reproducibility. Eq. (2) and Algorithm 1 refer to an object detection model Φ_D and an embedding model Φ_E, but the concrete models are never named, and no details are given for how object-wise embeddings are extracted for either VirtualHome or ALFWorld. Also, Algorithm 1 line 26 computes the current prototype p from a single state s_t, while the text describes 'trajectory-based' prototypes; it is unclear whether p uses one state, a history of states, or a full episode. These details are necessary to reproduce the method and to interpret the Wasserstein-distance retrieval. Please specify the exact models, the prototype computation from current observations, and the distance metric used in practice.","section":"Section 3.2 / Appendix C.2 / Algorithm 1"}],"minor_comments":[{"comment":"The claims '20.41%' and '26.58%' improvements are percentage-point differences (e.g., 45.71 to 66.12 in VirtualHome zero-shot). Please state 'percentage points' or give relative improvements, since the two readings differ substantially.","section":"Section 1 / Tables 1–2"},{"comment":"The caption 'Num. of WMs' is ambiguous: it could mean the number of selected models K or the total pool size N. Since the main setup uses K=3 from N=6, please clarify whether Table 5 varies K while holding N fixed or varies N.","section":"Table 5"},{"comment":"The table is labeled 'Multi-modal WorMI' and the text says 'We will include these experimental results in the final version.' A submitted paper should not contain placeholder text or incomplete experimental sections. Please finalize the results or remove them.","section":"Appendix D.2 / Table A.7"},{"comment":"The text of Figure 5 is garbled with '/uni00000031/uni00000052/...' tokens in the provided manuscript. Please check the source and regenerate the figure.","section":"Figure 5"},{"comment":"Minor typo: 'SayCayPay' appears in the first paragraph; should be 'SayCanPay'.","section":"Section 4.1"},{"comment":"The notation δ(E_j,E) treats Ej and E as objects with a Wasserstein distance, but Wasserstein distance is defined on probability measures. Please state explicitly that each embedding set is viewed as a uniform empirical measure (or, if the prototype definition is changed, as the corresponding weighted measure).","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is worth a revision cycle. The central empirical story is plausible and well-supported by ablations and confidence intervals, but the formal bound in Eq. (4) is incorrect as stated and the implementation details are incomplete. The placeholder text in Appendix D.2 also suggests the manuscript was submitted before being fully finalized. I recommend a major revision focused on fixing the theoretical claim and completing the experimental description, rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi colleague,\n\nWorth reading: WorMI is a clean recipe for composing small domain-specific world models with a frozen LLM planner at test time. The cross-attention composition of multiple world models plus the learned retrieval is new in an incremental way, and the experiments are thorough: two benchmarks, zero-shot and few-shot, ablations, sensitivity to LLM size, world-model count, complex instructions, even continual implanting/removal. The reported gains over SayCanPay are large (20-26% SR in VirtualHome) and the CIs on the main tables look small enough to take seriously. The attention-map analysis is more grounded than usual—it shows the module actually shifts weight onto the world model containing the relevant object or skill.\n\nThe soft spot you should know about: the theory in Section 3.2/Appendix A.1 is wrong. The triangle inequality argument for prototype retrieval relies on W(E_j, p_j) ≤ ρ, which does not follow from the k-center condition when p_j is given uniform mass over clusters. For imbalanced clusters, W can be arbitrarily large even at ρ=0. So the claim that prototypes are a \"bounded proxy\" for the empirical measure is false as stated. That doesn't sink the empirical contribution—the ablation (WorMI-P vs prototype) shows only a 0.4% SR loss from using prototypes—but it means the theoretical justification should be deleted or corrected. As written, the paper's own theorem is unsupported.\n\nOther issues are more standard: no code or data, episode generation is under-specified, and some analysis tables (especially appendix ones) lack error bars. The scalability analysis (N=12) shows the retrieval helps, which is good evidence that the method isn't just the cross-attention.\n\nOverall: a solid applied contribution that will be useful to people working on test-time adaptation for embodied agents. It deserves a serious referee if resubmitted, but I'd ask them to either fix or drop the theory section. For a reading group it's a fine example of how to do ablations with a new composition method.\n\nIf I were an editor, I'd send it to review and ask for the theory to be fixed. I would cite it if I needed a test-time composition baseline.","headline":"A practical, well-executed composition method for LLM-based embodied agents, hobbled by a bogus theory section that shouldn't mask the strong empirical work.","tokens_in":20416,"tokens_out":2715,"would_cite":true,"duration_ms":24514,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By retrieving a few small domain-specific world models and fusing them into a frozen LLM at test time, WorMI achieves large zero-shot and few-shot gains in embodied task success, improving over SayCanPay by 20–27 percentage points.","keywords":["test-time adaptation","world models","embodied agents","LLM composition","prototype-based retrieval","compound attention","zero-shot generalization","meta-learning"],"falsifier":"Construct two domains with very similar object inventories but different transition dynamics (e.g., same objects, opposite affordances) and measure whether WorMI's prototype retrieval still selects the functionally correct world model. If prototype distance ranks the two domains as nearly identical, the zero-shot advantage over random retrieval should disappear; the paper's own adversarial-ratio experiment (Table A.9) already shows a collapse to 38% success at a 50% adversarial ratio.","tokens_in":19458,"feed_emoji":"🤖","tokens_out":5963,"duration_ms":50950,"temperature":0.7,"pith_summary":"This paper argues that an embodied agent can adapt to new, unseen environments without retraining by implanting a handful of domain-specific world models into a frozen large language model at test time. The central claim is that a prototype-based retrieval step—selecting the most relevant world models from the similarity of object-level state embeddings—combined with a learned 'compound attention' module that fuses and aligns those models' representations with the LLM's reasoning, yields large gains in zero-shot and few-shot task success. In VirtualHome, the framework (WorMI) improves average success rate over the SayCanPay baseline by 20.41 percentage points zero-shot and 26.58 points few-shot, with comparable gains in ALFWorld. If correct, this suggests a scalable recipe: keep the reasoning model fixed, acquire small per-domain models, and compose them on demand rather than fine-tuning the whole policy.","feed_headline":"World-model implanting lifts embodied agents across unseen domains","feed_subtitle":"By selecting and fusing only the most relevant domain models, WorMI beats strong LLM baselines without retraining.","key_machinery":"The central mechanism is the 'world-wise compound attention' Cθ, a parameter-efficient module consisting of a linear projection, a world-level cross-attention layer, and a reasoning-level cross-attention layer. It takes the intermediate layer outputs of the K retrieved world models and the reasoning model's hidden state, and injects a fused, aligned representation into the next layer of the frozen LLM. The complementary machinery is the prototype-based retrieval: each domain's dataset is summarized by k cluster centers of object-wise embeddings, and at test time the agent retrieves the models whose prototype distance to the current observation is smallest under the Wasserstein metric. Togeth","core_discovery":"The discovery is that domain-specific world models—small models trained on transitions, affordances, and behavior cloning for one domain—can be treated as composable modules that plug into a frozen LLM-based policy. The framework's two-stage mechanism, world-to-world integration (via a world-level cross-attention that weighs the retrieved models) and world-to-reasoning alignment (via a reasoning-level cross-attention that conditions on the LLM's hidden states), is what lets the policy selectively use knowledge from multiple domains. The authors show that retrieval quality matters (random retrieval drops success from 66.1% to 62.0%, and using all models is worse than the retrieved three), and","pith_inferences":["The retrieval's reliance on object-level embeddings suggests a testable boundary: two domains with similar objects but opposite physical dynamics might be misranked, so a dynamics-aware prototype (e.g., including transition embeddings) could be a stronger, more general selector.","A learned retriever trained end-to-end with the compound attention could replace the Wasserstein distance and remove the reliance on the quality of the offline clustering bound; this is not explored in the paper.","The paper's adversarial-model experiment implies that the compound attention filters out bad models up to a point; a safety-oriented deployment might add an explicit validity check that verifies a retrieved model's next-state predictions against the current observation before trusting it.","The k-center approximation bound in Eq. (4) is loose in the worst case, so the theoretical guarantee is weaker than the empirical evidence; a tighter, distribution-aware bound could make the framework's reliability claims more rigorous."],"forward_implications":["Deployed agents can be given new capabilities by adding a small world model and retrieving it, without touching the base LLM's weights.","Because models can be removed as easily as added, the framework supports continual adaptation and machine unlearning of specific domains.","The number of simultaneously implanted models matters: the paper's experiments show peak performance at 2–4 models and a drop at 6, so selection is a real constraint, not a formality.","WorMI's advantage grows with the base LLM's size, indicating it leverages, rather than bypasses, the reasoning model.","The compound attention generalizes across model counts and even to a multimodal reasoning model with only a slight performance drop, per the appendix."],"supporting_citations":[{"why":"Provides the SayCanPay baseline, the strongest comparison that WorMI must beat in both environments.","marker":"(Hazra et al., 2024)"},{"why":"Supplies the VirtualHome simulation environment and household-task dataset used for the main evaluation.","marker":"(Puig et al., 2018)"},{"why":"Supplies the ALFWorld text-based embodied task environment used as the second evaluation benchmark.","marker":"(Shridhar et al., 2021)"},{"why":"Defines the zero-shot planner baseline ZSP that WorMI is compared against.","marker":"(Huang et al., 2022a)"},{"why":"Defines the LLM-Planner in-context learning baseline that WorMI contrasts with.","marker":"(Song et al., 2023)"},{"why":"Introduces CALM, the model-composition idea that WorMI extends to embodied agents with multiple world models.","marker":"(Bansal et al., 2024)"},{"why":"Supplies the first-order meta-learning algorithm used to train the compound attention module.","marker":"(Nichol, 2018)"},{"why":"Provides the CL-ALFRED data-clustering setting used to partition ALFWorld into scene and task domains.","marker":"(Kim et al., 2024)"}],"fun_headline_variants":["Implant world models into LLM policies for instant domain adaptation","WorMI plugs domain-specific models into frozen LLM, boosts zero-shot","Selective world-model implanting beats LLM baselines on new domains","Test-time world-model fusing lifts embodied agents without retraining","Composable world models adapt embodied agents at test time"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The retrieval step assumes that a small set of representative object-level prototypes of each domain preserves the true ordering of domain similarity, so that the most relevant world models are always selected; if the prototypes misorder domains, the framework's gains should vanish.","fun_headline_variants_meta":{"raw":{"variants":["Implant world models into LLM policies for instant domain adaptation","WorMI plugs domain-specific models into frozen LLM, boosts zero-shot","Selective world-model implanting beats LLM baselines on new domains","Test-time world-model fusing lifts embodied agents without retraining","Composable world models adapt embodied agents at test time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2703,"prompt_tokens":743,"completion_tokens":1960,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":1885}},"tokens_in":487,"tokens_out":1960,"duration_ms":13107,"temperature":1.0,"reasoning_tokens":1885,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:30:39.973278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct two domains with very similar object inventories but different transition dynamics (e.g., same objects, opposite affordances) and measure whether WorMI's prototype retrieval still selects the functionally correct world model. If prototype distance ranks the two domains as nearly identical, the zero-shot advantage over random retrieval should disappear; the paper's own adversarial-ratio experiment (Table A.9) already shows a collapse to 38% success at a 50% adversarial ratio.","supporting_citations":[{"cited_title":"Z., and De Raedt, L","cited_arxiv_id":null,"evidence_quote":"Provides the SayCanPay baseline, the strongest comparison that WorMI must beat in both environments."},{"cited_title":"Virtualhome: Simulating household activities via programs","cited_arxiv_id":null,"evidence_quote":"Supplies the VirtualHome simulation environment and household-task dataset used for the main evaluation."},{"cited_title":"Alfworld: Aligning text and embodied environments for interactive learning","cited_arxiv_id":null,"evidence_quote":"Supplies the ALFWorld text-based embodied task environment used as the second evaluation benchmark."},{"cited_title":"H., Wu, J., Washington, C., Sadler, B","cited_arxiv_id":null,"evidence_quote":"Defines the LLM-Planner in-context learning baseline that WorMI contrasts with."},{"cited_title":"LLM Augmented LLMs: Expanding Capabilities through Composition","cited_arxiv_id":"2401.02412","evidence_quote":"Introduces CALM, the model-composition idea that WorMI extends to embodied agents with multiple world models."},{"cited_title":"Online continual learning for interactive instruction following agents","cited_arxiv_id":null,"evidence_quote":"Provides the CL-ALFRED data-clustering setting used to partition ALFWorld into scene and task domains."}],"review_version":1}