{"id":"cbe54cf6-ab2d-4942-ae39-c81a291c6ac8","arxiv_id":"2412.00103","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A zero-shot person search system that uses GPT-4o to score rooms and pick waypoints, outperforming MDP, HMM, and random walk baselines in simulated and one real-world test.","lead":"MLLM-Search uses a multimodal large language model to plan where a robot should look for a person, using a map overlaid with numbered waypoints and room labels. It is tested in simulation and one real building, and reports faster searches than prior planners in scenarios where schedules are missing or changed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark confound: GPT-4o both generates the test scenarios and supplies the baseline planners' probabilities, so the reported SOTA advantage may reflect shared reasoning priors rather than the MLLM-Search architecture.","rationale":"The reader's weakest_assumption correctly identifies the benchmark validity problem: the same model family (GPT-4o) generates the test scenarios, provides the baseline probabilities, and implements the proposed planner. This is the most load-bearing point because the paper's central contribution is the claimed superiority over SOTA planners; if the comparison is confounded, the quantitative headline is not established. I agree with the reader's conditional verdict. An independent evaluation would settle the issue: generate scenarios with a different mechanism, initialize baselines with their intended inputs, and report variance. The real-world single-trial evidence is supportive but not sufficient to independently validate the simulation comparison. No ad hominem is intended; the concern is strictly about experimental design and reproducibility.","tokens_in":13366,"tokens_out":3647,"duration_ms":34798,"concrete_test":"Re-run the comparison on a held-out set of scenarios generated by a mechanism independent of GPT-4o (e.g., human-authored schedules or a different LLM family such as Claude or Gemini), and initialize the MDP/HMM planners with probabilities derived from those schedules or from the independent generator rather than from GPT-4o. Use at least 10 random trials per condition and report mean plus/minus standard deviation for SR, SPL, and ST. If MLLM-Search still dominates on all four schedule types, the concern is resolved; if its advantage shrinks or reverses on Complete or Shifted schedules, the headline claim should be downgraded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim is that MLLM-Search outperforms MDP/HMM/Random-Walk planners (Section V-F). The load-bearing assumption is that this comparison is fair, but the experimental protocol couples the test generator, the planner, and the baselines through the same model family. Section V-C states that GPT-4o generated all test scenarios 'given the waypoint map and object locations'; Section V-E states that GPT-4o also generated 'transition and user location probabilities for each scenario' for every baseline. Thus the distribution over target locations and the baselines' priors are both outputs of the same model that implements MLLM-Search's region and waypoint planning. If GPT-4o's world-knowledge prior assigns high probability to the same semantically plausible regions that MLLM-Search selects, the baseline planners are initialized with priors that already encode the answer distribution, and the comparison measures prior alignment rather than the online search policy. This is an internal threat to validity, not a mere consensus disagreement. The absence of per-condition variances or trial counts (Table I) makes it impossible to assess whether the 0.2-0.3 SR gaps are within noise. The real-world trial (Section VI) is one successful trajectory with no failure cases, so it cannot independently establish generalization; it is consistent with the concern rather than a decisive counterexample.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MLLM-Search, a zero-shot person-search architecture for mobile robots that uses a multimodal large language model (GPT-4o) to plan search regions and waypoints from a semantically and spatially grounded waypoint map. The map is generated by overlaying navigable waypoints and semantic region labels on an occupancy map, and the planner alternates between a region planner scoring rooms by likelihood, proximity, and recency, and a waypoint planner using a spatial chain-of-thought prompt. The authors validate the method in 3D Gazebo simulations in hospital and office environments under complete, shifted, partial/incomplete, and absent schedules, report ablations of the main design components, and describe one real-world trial in a university building. The central claim is that MLLM-Search outperforms MDP, HMM, and random-walk baselines in search efficiency (SR, SPL, ST).","tokens_in":13615,"tokens_out":4233,"duration_ms":52729,"significance":"If the results hold, this would be a meaningful step for robotic person search: it is the first system to apply MLLMs to this task, and the visual prompting of a global waypoint map is a plausible and useful way to give LLM-based planners spatial grounding. The two-stage region/waypoint decomposition with recency, proximity, and likelihood scores is well motivated, and the ablation study attempts to isolate contributions of each design choice. The system is also relatively practical, using off-the-shelf components for mapping, tracking, and navigation. However, the current evidence for the main comparative claim is weakened by the experimental protocol, which couples the scenario generator, the baseline probability inputs, and the proposed planner through the same model family, and by the absence of statistical characterization of the simulation results. The real-world evidence is limited to a single successful trial. These issues are fixable in a revision but are load-bearing for the paper's headline conclusion.","major_comments":[{"comment":"The benchmark comparison is internally confounded by using GPT-4o in three roles: as the generator of the test scenarios (Section V-C), as the provider of transition and user-location probabilities for the MDP and HMM baselines (Section V-E), and as the MLLM that implements MLLM-Search's region and waypoint planning. Because the target locations in the test scenarios are sampled from GPT-4o's semantic prior over the same waypoint map that MLLM-Search queries, and because the baselines are initialized with probabilities from that same model, the reported SOTA advantage may reflect prior alignment rather than the search policy. To make the comparison valid, please (a) generate test scenarios from an independent process or from actual schedule data with randomized real-time events, (b) estimate baseline transition and likelihood probabilities from the actual schedule ground truth or a held-out training set rather than from GPT-4o, and (c) include a condition in which target locations are drawn uniformly to test whether MLLM-Search's advantage persists when semantic priors are not informative. Without these changes, the central claim of Section V-F ('our MLLM-Search method outperformed the SOTA planners') is not adequately supported.","section":"V-C, V-E"},{"comment":"No trial counts, variances, confidence intervals, or statistical tests are reported for any cell in Table I. The SR differences between MLLM-Search and the MDP/HMM baselines are often 0.2–0.3 (e.g., Complete Schedule office: MLLM-Search 1.00 vs. HMM 0.90; No Schedule hospital: 0.80 vs. HMM 0.50), and without knowing the number of trials per cell these gaps could easily be within sampling noise. The text further makes strong quantitative claims such as 'up to 193% improvement in SPL' (Section V-F) that cannot be evaluated without error bars. Please report the number of trials per scenario, mean ± standard deviation or 95% confidence intervals, and a paired significance test for at least the main MLLM-Search-versus-baseline comparisons.","section":"Table I, V-F"},{"comment":"The real-world evaluation consists of a single successful trajectory with no repeated trials, no failure cases, and no variation in user or environment parameters. The conclusion in Section VII that MLLM-Search 'was able to generalize' to a new environment overstates what can be concluded from one positive example. Please provide multiple real-world trials, ideally with different users, schedule types, start locations, and target movement patterns, and report both success rate and failure cases. At minimum, the wording should be softened to describe the trial as a demonstration rather than a generalization result.","section":"VI"}],"minor_comments":[{"comment":"The SPL equation is garbled in the text (it reads 'MZ∑𝑆\\'Z\\'OMU![B\\(V!,U!)'). Please restate it with clear definitions of S_i, l_i, and p_i.","section":"V-B"},{"comment":"In Eq. (4), the set 'I' used in the argmin is not defined; presumably it refers to the set of all points in the occupancy map or the set of candidate safe points, but this should be stated explicitly.","section":"IV-A.2, Eq. (4)"},{"comment":"Eq. (1) lists image x_t as an input to the action function f, but the later architecture description treats the semantic and waypoint maps as the main spatial inputs. Please clarify the role of x_t in the formulation and whether it is the current camera image or a placeholder for the robot's observation.","section":"III-A, Eq. (1)"},{"comment":"The schedule-type terminology is inconsistent: Section V-C lists 'Partial/Incomplete Schedules' as one category but then distinguishes partial schedules (1–2 hour gaps) from incomplete schedules (larger gaps). Please align the naming across the text, Table I, and the scenario descriptions.","section":"V-C"},{"comment":"Reference [6] for LDTrack is cited as 'we have developed' but appears only as an arXiv preprint; if a peer-reviewed version exists, it should be cited instead.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the core architecture is creative, but the benchmark design is the main weakness. The authors should be encouraged to either re-run the comparison with properly calibrated baselines and an independent scenario generator, or substantially temper the superiority claim. I would also encourage the editors to ask for code or detailed experimental logs, since GPT-4o-based planning is stochastic and the paper currently provides no release of scenarios, prompts, or trajectories, which will make independent replication difficult."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a sensible system paper with an evaluation design that undercuts its own headline claim. The architecture is genuinely new in combining visual prompting of a global waypoint map with an MLLM region/waypoint planner and a spatial chain-of-thought. The ablation is well structured and mostly supports the design choices; the SCoT variant shows clear time savings and the score ablations degrade in the expected directions. The real-world trial is a nice proof-of-concept, though it is one success case.\n\nThe soft spot is the benchmark. Section V-C says GPT-4o generated the test scenarios from the same waypoint map and object locations the planner sees. Section V-E says GPT-4o also generated transition and user-location probabilities for every baseline. So the same model family that implements MLLM-Search writes the test distribution and initializes the MDP/HMM priors. If GPT-4o's semantic prior aligns with what MLLM-Search selects, the comparison partly measures prior alignment rather than the online policy. That is an internal threat to validity, and the paper does not acknowledge it. It also makes the baseline comparison uneven: MDP and HMM are normally trained on historical schedule data, not handed probabilities from a language model; giving them GPT-4o priors is neither their standard setup nor a clean control.\n\nThe absence of variance or trial counts in Table I is a lesser but real issue. With no repeats reported, I can't tell whether the 0.2-0.3 SR gaps are stable. The single real-world run is consistent with the claim but cannot establish generalization on its own. The related work also leans heavily on the authors' own prior papers, which is natural in this niche but means the novelty claim is only checked against a small slice of the literature.\n\nThe central idea is plausible and the paper is well written. The method is described in enough detail to reproduce. But the headline 'outperforms SOTA' should be treated as conditional until the benchmark is rerun with scenarios generated independently (or held out from the planner's model family) and baselines given their proper historical inputs, and with repeated trials.\n\nWho this is for: people working on robot search, semantic navigation, and MLLM planning. A serious referee should see it; the architecture and the confound are both worth airing. My recommendation: send to peer review, but the reviewers should push hard on the evaluation protocol.","headline":"Useful architecture, but the headline comparison is compromised by using GPT-4o to generate scenarios and initialize baselines with the same reasoning priors.","tokens_in":14148,"tokens_out":2587,"would_cite":true,"duration_ms":20723,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MLLM-Search claims that a vision-language model prompted with a labeled waypoint map can plan a robot's search for a moving person with no schedule data, and beat MDP, HMM, and random-walk planners in event-driven scenarios.","keywords":["robotic person search","multimodal large language models","zero-shot search","event-driven scenarios","visual prompting","semantic mapping","waypoint planning","mobile robots"],"falsifier":"Re-run the hospital and office benchmarks with test scenarios and MDP/HMM transition-and-likelihood tables built from independently observed movement logs, with no GPT-4o involved in data generation, and count whether MLLM-Search still leads on success rate and SPL in the no-schedule conditions; if the margin shrinks to ablation-level noise, the advantage is partly a benchmark artifact. Separately, run the real-world protocol in several buildings with several users and measure how often the robot beats a random-walk baseline, since the paper's generalization claim rests on a single successful trial.","tokens_in":13165,"feed_emoji":"🤖","tokens_out":23767,"duration_ms":176054,"temperature":0.7,"pith_summary":"Robots that must find a specific person—a resident in a care home, a doctor in a hospital, a visitor in an office—usually need a model of that person's habits: a daily schedule, past locations, or a last known position. MLLM-Search's claim is that this requirement can be dropped: a multimodal large language model (here GPT-4o) can plan the search from scratch each time by looking at a waypoint map of the building whose rooms and objects are labeled, plus the search query and any available context such as room bookings or a partial schedule. The architecture splits planning into a region planner, which scores each room by likelihood, proximity to the robot, and recency of past searches, and a waypoint planner, which uses a spatial chain-of-thought prompt to route the robot past semantically relevant objects. In photorealistic simulations with complete, shifted, partial, or absent schedules, the paper reports that this approach beats Markov decision process (MDP), hidden Markov model (HMM), and random-walk planners on success rate and search efficiency, with the largest margins precisely when schedule information is missing. A single real-world trial in a university building is offered as evidence that the method transfers to an unseen environment.","feed_headline":"No-schedule robot search finds people that probability planners miss","feed_subtitle":"The robot reads a labeled room map and reasons where someone would be, so no schedule data is required","key_machinery":"The load-bearing object is the semantically and spatially grounded waypoint map: a topological graph of navigable waypoints placed on the occupancy grid, with each waypoint numbered and each region labeled with a semantic name, rendered as a single image prompt. This visual prompt is what turns the MLLM's weak native spatial reasoning into usable global spatial reasoning, because coordinates and labels are visible in the image instead of being left to the model's imagination. Two mechanisms do the planning work around it: the region planner selects the next region by the summed scores $s_l + s_p + s_r$ (likelihood, proximity, recency) under chain-of-thought prompting, and the waypoint planner uses spatial chain-of-thought (SCoT) prompting, which decomposes path choice into per-waypoint steps listing the candidate next waypoints and the objects at each, to rank candidate paths generated by A*. The single-stage variant that merges both planners performs markedly worse, which the paper attributes to long-context degradation.","core_discovery":"The central claim, stated on the paper's own terms, is that a pretrained multimodal language model can act as a zero-shot person-search planner if the environment is handed to it in the right visual form. The map generation subsystem converts a metric map into a semantically and spatially grounded waypoint map: numbered navigable waypoints, computed from the distance transform of the occupancy grid and clustered with k-means, overlaid with text labels for regions such as 'Main Lobby' or 'ICU.' The region planner then gives each region a likelihood score, a proximity score, and a recency score, and the robot moves to the highest-sum region; the waypoint planner ranks A*-generated candidate paths by the objects each waypoint passes, using spatial chain-of-thought prompting so the model reasons about local geometry step by step. The paper's claim is that search degrades gracefully as schedules become incomplete or disappear, because the model substitutes semantic and contextual inference—'a doctor during an emergency is probably in the ICU or operating room'—for the probability tables that MDP and HMM planners must derive from history.","pith_inferences":["My inference: the architecture is a general template rather than a person-finder only—any query of the form 'where would someone plausibly be given these events' (missing equipment, lost visitors, post-disaster victims) could reuse the same prompted waypoint map with a different object list.","The paper does not test whether the reported gains come from GPT-4o's specific priors or from the prompting structure itself; re-running the same architecture with an open-weight MLLM would separate the two.","The single real-world trial is anecdotal on its own; in my reading the simulation results carry the claim, and a multi-building, multi-user deployment study would be needed to know how well the zero-shot advantage transfers outside photorealistic simulators.","A natural extension the authors do not run: apply the same two-stage prompted planner to searching for objects rather than people, since the waypoint planner already ranks paths by object relevance and only the target-matching module would change."],"forward_implications":["A robot can start searching for a person in a building it has never searched before, with no training data about that person, as long as a metric map and a semantic map of rooms and objects exist.","Search quality holds up when real-time events shift a person's schedule; the largest measured gains over MDP, HMM, and random-walk baselines come in no-schedule scenarios, where the paper reports up to 50% higher success rate, up to 193% higher success-weighted path length (SPL), and search times as much as 22.4 minutes faster.","Splitting planning into region-then-waypoint stages is load-bearing: the single-stage variant loses up to 40 percentage points of success rate in no-schedule scenarios, consistent with long-context reasoning failures.","Each score component in the region planner earns its place: dropping recency causes repeated room visits and search times up to about 21 minutes longer, dropping proximity cuts success rate by up to 40%, and dropping likelihood costs up to 30% success.","Contextual reasoning stands in for user models: from a query like 'deliver supplies to the doctor during an emergency,' the planner infers ICU, operating room, or exam room without any personal history."],"supporting_citations":[{"why":"HMM-based Casper planner; the probabilistic lookahead baseline that MLLM-Search is measured against.","marker":"[3]"},{"why":"MDP planner requiring user location PDFs; one of the decision-theoretic baselines in the comparison study.","marker":"[20]"},{"why":"Schedule-based MDP search method for multi-resident settings; supplies the reward modeling that the comparison extends to event-driven scenarios.","marker":"[21]"},{"why":"GPT-4o, the multimodal language model that performs all region and waypoint reasoning and also generated the test scenarios and baseline probabilities.","marker":"[28]"},{"why":"PIVOT visual prompting; the prior method whose coordinate-overlay idea is adapted from single scenes to a whole-environment waypoint map.","marker":"[34]"},{"why":"Retrieval-augmented generation; pulls the relevant schedule and database chunks into the planner's context window.","marker":"[41]"},{"why":"Chain-of-thought prompting; the base prompting method that the region planner uses and that spatial chain-of-thought extends.","marker":"[42]"},{"why":"A* search; generates the candidate waypoint paths that the waypoint planner ranks by object relevance.","marker":"[43]"},{"why":"Lost in the middle; cited as the reason merged single-stage planning degrades because long contexts hurt MLLM performance.","marker":"[50]"}],"fun_headline_variants":["Zero-shot robot search reads room maps to find people without schedules","Robot uses language model to find people without any schedule info","Spatial LLM reasoning finds people with zero schedule data","LLM reads room maps to find people even with no schedule info"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark is unbiased: the same model, GPT-4o, that runs MLLM-Search also generated the test scenarios and the probability tables for the MDP and HMM baselines, so the reported lead presumes that shared origin does not favor the proposed method.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot robot search reads room maps to find people without schedules","Robot uses language model to find people without any schedule info","Spatial LLM reasoning finds people with zero schedule data","LLM reads room maps to find people even with no schedule info"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000823,"raw_usage":{"total_tokens":3662,"prompt_tokens":1072,"completion_tokens":2590,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":2519}},"tokens_in":688,"tokens_out":2590,"duration_ms":17862,"temperature":1.0,"reasoning_tokens":2519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:53:03.531158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the hospital and office benchmarks with test scenarios and MDP/HMM transition-and-likelihood tables built from independently observed movement logs, with no GPT-4o involved in data generation, and count whether MLLM-Search still leads on success rate and SPL in the no-schedule conditions; if the margin shrinks to ablation-level noise, the advantage is partly a benchmark artifact. Separately, run the real-world protocol in several buildings with several users and measure how often the robot beats a random-walk baseline, since the paper's generalization claim rests on a single successful trial.","supporting_citations":[],"review_version":1}