{"id":"9aaf2f0d-61f5-4394-b356-d29b2274caeb","arxiv_id":"2603.23800","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-estimated object-location probabilities plus map costs yield a model-based planner that beats pure-LLM and optimistic search, while offline replay selects prompts/LLMs faster than UCB.","lead":"Robots find household objects faster when an LLM only estimates where items are likely to be, and a classical planner then chooses the search order using those estimates plus map travel costs. The same high-level search abstraction also lets the robot pick the best prompt and LLM at deployment time via offline replay, beating standard bandit selection.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged calibration premise; the empirical claims hold under the paper's own evidence.","rationale":"The strongest claim is a pair of measured empirical improvements (LLM+MODEL vs LLM-DIRECT / optimistic, and Replay vs UCB) under a shared high-level abstraction. The reader's weakest assumption correctly isolates the place where the argument could fail: if the LLM numbers are pure noise, the model-based planner should not systematically beat a pure-LLM selector that receives the same semantic information. The paper already supplies the relevant evidence: multiple prompts, two proprietary and two open-source LLMs, 150 ProcTHOR maps, and a small real-apartment suite all show the same direction. Missing error bars and the tiny real-robot N keep the verdict conditional, exactly as the reader concluded; they do not introduce a new load-bearing flaw. The concrete test above would cleanly separate 'LLM probabilities carry useful ordinal signal' from 'map costs alone explain the gains.' Until that (or an equivalent ablation) is run, the existing evidence is sufficient to leave the reader's CONDITIONAL / HIGH verdict unchanged.","tokens_in":8672,"tokens_out":572,"duration_ms":7168,"concrete_test":"Recompute the LLM+MODEL rows of Table 1 after replacing each LLM probability vector with its rank-order permutation (or with a uniform distribution over containers) while keeping travel costs fixed; if the cost advantage over LLM-DIRECT disappears or reverses, the planner is relying on ordinal/cardinal information that is not robust. If the advantage remains, the map-cost component alone is doing most of the work and the calibration premise is weaker than claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (that LLM scalar probabilities are sufficiently well-calibrated or ordinally useful to parameterize an expected-cost search model) is the correct load-bearing premise, but it is not a soft spot that undermines the central claim as stated. The paper never claims general calibration; it claims that, when those LLM outputs are plugged into a high-level expected-cost planner that also uses map travel costs, the resulting policy beats pure-LLM action selection and optimistic greedy search on ProcTHOR (150 maps) and on a real apartment. Tables 1 and 3, the open-source LLM results, and the five real-robot trials all show consistent directional gains for LLM+MODEL over LLM-DIRECT. Because the claim is comparative and empirical rather than a claim of absolute calibration, the premise is already stress-tested by the reported experiments. No additional internal inconsistency or hidden assumption that would reverse the ranking is visible in the provided text.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes an LLM-informed model-based planner for object search in partially-known household environments: an LLM supplies per-container discovery likelihoods that, together with map-derived travel costs, instantiate a high-level expected-cost search model (LLM+MODEL). The same action abstraction is then used for deployment-time prompt/LLM selection via offline replay, compared against UCB. On 150 ProcTHOR maps the approach reports up to 11.8% lower average navigation cost than a fully LLM-driven selector (LLM-DIRECT) and up to 39.2% lower than optimistic greedy search; open-source LLM ablations and a five-trial apartment deployment show the same direction. Replay selection is reported to cut average cost by 6.5% and cumulative regret by 33.8% relative to UCB over 100-trial selection runs.","tokens_in":8880,"tokens_out":1163,"duration_ms":26573,"significance":"If the empirical ranking holds under proper statistical reporting, the work is a useful, practical contribution to embodied LLM planning: it shows that treating LLM outputs as model parameters inside a classical high-level planner can outperform both pure LLM action selection and uninformed search, and that the same abstraction enables fast offline-replay prompt/LLM selection without on-robot trial-and-error. Strengths include multi-LLM evaluation (GPT-5 Mini, Gemini 2.5 Flash, GPT-OSS, Llama3.2), consistent directional gains, a real LoCoBot apartment suite, and an explicit link to prior offline-replay model selection. The free parameters (hand-designed prompts, selection horizon) are disclosed. The result is incremental rather than foundational, but it is actionable for household object search and for deployment-time model selection in LLM-robot systems.","major_comments":[{"comment":"Table 1 and Sec. 6.2 report only mean navigation costs over 150 maps, with no standard errors, confidence intervals, or hypothesis tests. The headline claims (up to 11.8% vs LLM-DIRECT, 39.2% vs OPTIMISTIC+GREEDY) are therefore hard to assess for statistical reliability; the same holds for the open-source results in Table 3. With N=150 already collected, means should be accompanied by SE/CI (or bootstrap) and pairwise significance so the percentage improvements can be treated as load-bearing evidence rather than point estimates.","section":null},{"comment":"Sec. 6.3 and the selection results (including the real-apartment Figure 7) likewise give average cost and cumulative regret without variance or trial-level trajectories with error bands. The claimed 6.5% cost and 33.8% regret gains of Replay Selection over UCB rest on these aggregates; reporting per-trial regret curves with uncertainty (and the UCB exploration constant used in Eq. (2)) is needed to support the selection contribution.","section":null},{"comment":"The central modeling premise (Secs. 3–4 / LLM+MODEL) is that LLM scalar probabilities are useful parameters of an expected-cost search model. The paper never analyzes calibration or even rank correlation between those probabilities and empirical discovery rates on held-out maps. A short diagnostic (e.g., reliability diagram or Spearman correlation of LLM p vs. observed hit rates per container type) would show whether the planner is using well-ordered likelihoods or merely a soft ranking, and would strengthen the claim that the LLM is truly 'informing' the model rather than acting as a black-box prior.","section":null}],"minor_comments":[{"comment":"Real-apartment evaluation (Table 2, five target objects) is useful as a sanity check but is too small to support strong claims of 'similar improvements'; frame it explicitly as qualitative validation and avoid percentage comparisons that mirror the large-N sim results.","section":null},{"comment":"Prompt templates (P-CONTEXT-A/B, P-MINIMAL, P-DIRECT) are described in Sec. 6.1 and Appendix A, but the main text should state more clearly which information (room–container lists, distances) is available to the LLM at query time versus what is known only to the planner, to avoid conflating LLM-DIRECT with a true planner.","section":null},{"comment":"Figure 4 trajectories are illustrative; adding a short caption note on map identity and target object would make the qualitative comparison reproducible.","section":null},{"comment":"Token/cost Table 4 is helpful; consider also reporting approximate wall-clock LLM latency per query, since that affects real-time deployment of LLM+MODEL versus LLM-DIRECT.","section":null},{"comment":"Minor typography: spacing in titles ('OBJECTSEARCH', 'VIALLM-INFORMED') and occasional missing spaces around LLM acronyms should be cleaned for camera-ready.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The extract has large blank stretches where core method equations (planning objective, offline-replay scoring) are only referenced (Eqs. 2 and 4); if the submitted PDF is similarly thin on formal model definition, that would push me toward major_revision. Assuming a complete PDF with those sections, minor_revision for statistics and a calibration diagnostic is appropriate. Scope fits a robotics journal well; novelty is the combination of LLM likelihoods + map costs + offline-replay prompt selection rather than any single component."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean, usable result for household object search. The authors stop treating the LLM as a next-action oracle and instead ask it only for discovery likelihoods over named containers, then fold those numbers into a classical high-level expected-cost planner that already knows map travel costs. That combination beats both pure LLM-DIRECT selection (up to 11.8%) and optimistic greedy (up to 39.2%) across 150 ProcTHOR maps, holds for open-source models, and shows the same direction in a five-trial apartment run. The second contribution is equally practical: because every candidate policy shares the same high-level action abstraction, offline replay lets them rank prompts and LLMs faster than UCB, cutting average cost and cumulative regret.\n\nWhat is new is not “LLM for object search” (plenty of that already) but the strict separation of roles—LLM supplies model parameters, planner optimizes—and the demonstration that the same abstraction immediately supports deployment-time selection. Tables 1 and 3, the open-source ablations, and the real-robot numbers all point the same way; the comparative claim is therefore well supported by the evidence they actually report.\n\nSoft spots are real but secondary. Main tables give only means, no error bars or significance tests. Real-robot evidence is five trials. Prompts are hand-crafted and fixed. No code release. The load-bearing premise—that LLM scalars are at least ordinally useful inside an expected-cost model—is never claimed as general calibration and is already stress-tested by the head-to-head results; it does not reverse the ranking under the paper’s own data. Citations look appropriate; self-cites to their earlier offline-replay work are on-topic.\n\nThis is for people building embodied search systems who want a recipe they can implement tomorrow, not for foundational theory. It deserves a serious referee. I would send it out.","headline":"Solid empirical robotics paper: LLM likelihoods as planner parameters beat pure-LLM action selection, and the same abstraction speeds prompt selection via offline replay.","tokens_in":9557,"tokens_out":486,"would_cite":true,"duration_ms":6818,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"LLM estimates of object-finding odds, fed into a cost-aware planner with map travel costs, beat pure LLM action selection for household object search, and the same abstraction speeds prompt and model selection at deployment.","keywords":["object search","large language models","model-based planning","prompt selection","partially-known environments","offline replay","robot navigation"],"falsifier":"On a held-out suite of household maps and objects, insert the same LLM probability estimates into the model-based planner and measure average navigation cost; if that cost is no lower (or is higher) than the cost obtained by simply letting the identical LLM directly pick the next container, the claim that model-based integration improves search is falsified.","tokens_in":9536,"feed_emoji":"🤖","tokens_out":957,"duration_ms":19837,"temperature":0.7,"pith_summary":"This paper argues that robots searching for objects in partially mapped homes perform better when a large language model is asked only for likelihoods of finding the target in each container, and those numbers are combined with known travel costs inside a classical high-level planner, rather than letting the language model itself choose the next place to look. On simulated household maps the hybrid approach lowers average navigation cost by as much as 11.8 percent versus a pure language-model selector and by nearly 40 percent versus an uninformed optimistic baseline; real-apartment trials show the same pattern. Because the planner works with abstract “search this container” actions, past trajectories can be replayed offline, letting the robot select the best prompt and language model far faster than ordinary multi-armed bandit methods. A reader who wants reliable object search cares because pure language-model planners often act myopically or mishandle quantitative trade-offs, while this method keeps their commonsense knowledge yet restores long-horizon cost-aware reasoning.","feed_headline":"LLM odds plus map costs beat pure LLM search by 12%","feed_subtitle":"Same high-level actions also pick the best prompt and model faster than bandits, in sim and on a real robot.","key_machinery":"LLM-informed model-based planning (LLM+MODEL): the LLM is queried solely for per-container discovery probabilities; those values plus travel costs instantiate a model that a planner optimizes over high-level “search container” actions. The identical abstraction enables offline-replay selection of the best policy-prompt-LLM triple.","core_discovery":"An LLM can supply scalar discovery likelihoods for a target object inside named containers; when those likelihoods are combined with map-derived travel costs and inserted into a high-level expected-cost search model, the resulting planner outperforms both a pure LLM that directly chooses the next container and an optimistic greedy baseline. The same high-level action abstraction further permits offline replay of past experience, enabling faster deployment-time selection among candidate prompts and LLMs than standard UCB bandit selection.","pith_inferences":["The same probability-plus-cost model can be extended to multi-object or multi-robot search without redesigning the selection machinery.","If LLM probabilities remain poorly calibrated on rare objects, a lightweight online update from failed searches would further cut cost.","Any high-level abstraction that separates perception from long-horizon cost can turn foundation models into selectable modules rather than fixed policies.","Automated prompt generation, listed as future work, could close the remaining gap between hand-written prompts and an adaptive system that invents better ones."],"forward_implications":["Robots achieve lower average navigation cost for object search by letting LLMs parameterize a planner rather than replace it.","Prompt and LLM choice can be optimized at deployment without expensive on-robot trial-and-error, using offline replay of the high-level abstraction.","Performance gaps between prompts and LLMs are large enough that automatic selection matters more than fixing one prompt in advance.","The same hybrid framework transfers from simulation to a physical apartment robot with comparable relative gains.","Open-source LLMs also benefit, sometimes by larger relative margins than proprietary models."],"fun_headline_variants":["LLM likelihoods plus map costs beat pure LLM object search by 11.8%","Map travel costs fuse with LLM object odds for 12% better search plans","Model-based search using LLM stats and maps cuts cost 12% vs pure LLM","LLM odds plus map costs top pure LLM and optimistic baselines in search","Offline replay selects best LLM prompts faster than UCB, 6.5% lower cost"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The language model’s numerical guesses about how likely an object is to sit inside each named container are accurate enough, or at least ranked well enough, to serve as useful parameters inside an expected-cost planner.","fun_headline_variants_meta":{"raw":{"variants":["LLM likelihoods plus map costs beat pure LLM object search by 11.8%","Map travel costs fuse with LLM object odds for 12% better search plans","Model-based search using LLM stats and maps cuts cost 12% vs pure LLM","LLM odds plus map costs top pure LLM and optimistic baselines in search","Offline replay selects best LLM prompts faster than UCB, 6.5% lower cost"]},"model":"grok-4.5","effort":"low","cost_usd":0.006058,"raw_usage":{"total_tokens":1531,"prompt_tokens":782,"num_sources_used":0,"completion_tokens":107,"cost_in_usd_ticks":60580000,"prompt_tokens_details":{"text_tokens":782,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":642,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":782,"tokens_out":107,"duration_ms":6985,"temperature":1.0,"reasoning_tokens":642,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T19:20:59.919339+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out suite of household maps and objects, insert the same LLM probability estimates into the model-based planner and measure average navigation cost; if that cost is no lower (or is higher) than the cost obtained by simply letting the identical LLM directly pick the next container, the claim that model-based integration improves search is falsified.","supporting_citations":[],"review_version":1}