{"id":"84fde712-363c-40a1-8287-6301ed622f4d","arxiv_id":"2501.04947","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"SwPC uses conformal prediction to convert VLM similarity scores into calibrated candidate room sets, reporting improved success-versus-help trade-offs on Matterport3D.","lead":"This paper wraps a vision-language model in a statistical method called conformal prediction to output a short list of candidate rooms instead of a single guess, asking a human for help only when the list has more than one room. It reports better success at the same rate of human help than prompting baselines on Matterport3D, but the test descriptions are generated by a model, not real users.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The coverage guarantee and help-rate advantage are demonstrated only for LLaVA-generated descriptions, not genuine user language; since the motivating scenario is human ambiguity, the exchangeability assumption behind Eq. (4) is unverified for actual deployment.","rationale":"The paper's central claim is that SwPC provides statistically guaranteed prediction sets and, at the same success rate, requires less human help than baselines. The conformal guarantee is conditional on exchangeability between calibration and test data. The paper explicitly restricts its evaluation to LLaVA-generated descriptions, noting that 'the test data and calibration data follow the same distribution' because both are synthetic. This makes the reported coverage and help-rate curves valid for a distribution of machine-written descriptions, but the paper's motivating scenarios are human instructions ('got to the bedroom' with multiple bedrooms, ambiguous directions). No human-written instructions are collected or tested, so the core assumption behind Eq. (4) is unverified for the deployment condition. This is not merely a matter of 'changing the prompt style'; it is a distribution shift in the input text that can invalidate the conformal guarantee entirely. I agree with the Pith reader that this is the weakest assumption. The additional concerns (the fine-tuning contradiction in the calibration dataset collection, the notation errors in Eqs. (5)–(6), and the absent code/data) are real but secondary; they weaken the presentation and reproducibility, whereas the exchangeability gap directly undermines the validity of the central claim for real users. The concrete test I propose would settle the question: if human-instruction coverage remains at the nominal level and the help-rate comparison persists, the concern is resolved; if not, the rejection stands. Given the missing validation of the motivating condition, keeping the reader's REJECT verdict is appropriate.","tokens_in":7293,"tokens_out":7495,"duration_ms":78035,"concrete_test":"Run SwPC on a held-out set of human-generated instructions: ask at least 20 participants to write natural-language room requests (e.g., 'take me to the bedroom') for the same 1504 Matterport3D test rooms, use the LLaVA-based calibration set exactly as in the paper, set alpha=0.1, and compute empirical coverage and help rate. If the 95% Clopper-Pearson interval for coverage excludes 0.9, or if the success-vs-help curve at a matched success rate lies above the Fig. 3b CP curve, the exchangeability assumption fails and the real-world claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that calibration and test inputs be exchangeable (Theorem 1, Eq. (4)). The paper asserts this only because 'the test data and calibration data follow the same distribution,' where both are LLaVA-generated descriptions of Matterport3D rooms. This makes the Fig. 3 curves internal to the synthetic description distribution: the conformal guarantee holds for LLaVA-style text under repeated sampling, but the paper's motivation is precisely that human instructions carry ambiguity that VLMs may misclassify. No experiment uses instructions written by human users, so the evaluation never exercises the distribution the framework is claimed to protect. For real user language, coverage can drop below 1−alpha by an arbitrary amount; the stated guarantee gives no bound. Therefore the headline result (higher success at the same help rate, with coverage near 1−alpha) is unsubstantiated for the intended use, even if the CP machinery is internally correct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SwPC, a conformal-prediction wrapper around VLM cosine-similarity scores for room-level place recognition in indoor environments. The authors construct a calibration set from LLaVA-generated descriptions of Matterport3D top-down views, compute non-conformity scores as 1 minus the similarity score, and form prediction sets that are claimed to satisfy a coverage guarantee while minimizing requests for human help. Experiments compare SwPC against Prompt Set, Binary Set, and No Help baselines, reporting success-rate / help-rate trade-off curves that favor SwPC.","tokens_in":7465,"tokens_out":4039,"duration_ms":39459,"significance":"The application of split conformal prediction to VLM-based place recognition is a reasonable and lightweight idea, and the motivation for uncertainty-aware assistance in assistive robotics is well taken. The paper is clearly written in parts and names a concrete failure mode (VLM hallucination and ambiguous human instructions). However, the central empirical claim is not supported as presented: the evaluation is conducted entirely on LLaVA-generated descriptions, not on human language, and the conformal prediction machinery contains errors in the set construction and calibration-data definition that affect the reported curves. If the formulation is corrected and validated with genuine user input, the framework could be a useful baseline for the community, but the current manuscript does not yet deliver that evidence.","major_comments":[{"comment":"The prediction set definition includes an extra element: C(X_test) = {Y_{π_1}, ..., Y_{π_k}} with k = sup{k' : s(X_test,Y_{π_k'}) ≤ \\hat q} + 1. This adds one class even when no score falls below the threshold, so the set is not the standard conformal set {Y : s(X_test,Y) ≤ \\hat q}. The extra element systematically inflates both the prediction-set size and the probability of containing the true label, and it distorts the help-rate comparison in Fig. 3 because help is triggered by set size > 1. The authors should use C(X) = {Y : s(X,Y) ≤ \\hat q} and re-run all experiments.","section":"Eq. (9)"},{"comment":"The calibration dataset is not defined coherently. Eq. (5) builds D over all pairs (X_i, Y_j) for j = π_1(X_i),...,π_n(X_i), but Eq. (6) then restricts to j = π_i(X_i), which is the i-th position in the permutation rather than a class label. The phrase \"We include classes with true labels\" does not clarify how the true label's score is selected. As written, the scores in the calibration set are ambiguous, so the quantile in Eq. (8) is ill-defined and Theorem 1 cannot be applied to the implemented procedure. A precise definition of the non-conformity score for each calibration sample (e.g., s(X_i,Y_i) = 1 - f(X_i,Y_i) for the true label, or a rank-based score) is required.","section":"Eqs. (5)-(6), Calibration Dataset Collection"},{"comment":"The evaluation is circular with respect to the motivating application. The paper states \"Since we are generating descriptions for these rooms and using the same VLM model to generate the similarity score, the test data and calibration data follow the same distribution.\" Both calibration and test inputs are LLaVA-generated descriptions of the same rooms, so the coverage and help-rate results in Fig. 3 hold only for the distribution of LLaVA-style text, not for human instructions. The paper's motivation is precisely human ambiguity (e.g., \"go to the bedroom\"), and no experiment uses instructions written by human users. Consequently, the central claim that SwPC reduces human help at a given success rate for real assistive-robot interactions is unsubstantiated. The authors should validate exchangeability with human-collected or human-annotated instructions, or at minimum demonstrate robustness to realistic language perturbations.","section":"Experiment and Results, Dataset paragraph"},{"comment":"The quantile definition contains a malformed set expression: |{i : 1− f(X_i,Y_i)}≤ q| should read |{i : 1− f(X_i,Y_i) ≤ q}|. As printed, the set comprehension is incomplete, making the formal statement of the coverage guarantee unintelligible. This is a load-bearing equation because it defines the threshold \\hat q used throughout the method.","section":"Theorem 1, Eq. (2)"}],"minor_comments":[{"comment":"The upper bound 1−α+1/(n+1) in Eq. (1) is the usual finite-sample bound for a specific quantile choice; the definition in Eq. (2) uses ⌈(n+1)(1−α)⌉/n, and the authors should verify that the stated bound holds for this particular choice or cite the exact version from the conformal prediction literature.","section":"Eq. (1)"},{"comment":"The notation is inconsistent: the text introduces \"user Y_i ∈ Y\" and \"room images X_i ∈ X\", but then uses (X_i,Y_i) with Y as the label; this should be cleaned up to avoid confusing the reader.","section":"Preliminary Knowledge"},{"comment":"The Gibbs and Candès reference is cited as 2024 NIPS '21, which appears to be incorrect; the bibliographic entry should be checked and updated.","section":"References"},{"comment":"The term \"Quartile\" is used where \"quantile\" is meant; this is more than a spelling issue because the meaning of the mth order statistic should be precise.","section":"Eq. (8) and text"},{"comment":"The paper does not state the size n of the calibration set, the number of scenes used for calibration versus fine-tuning, or any variance/confidence intervals across the 43 test scenes; these details are necessary to assess the finite-sample guarantee claimed in Eq. (1).","section":"Experiment and Results"},{"comment":"The caption says \"Binary and No Help Set are not shown on the left since prediction sets are not provided,\" but the figure itself has no such panels; the caption should be aligned with the actual figure content.","section":"Fig. 4 caption"}],"recommendation":"reject","confidential_remarks":"The reviewer's concern about the self-referential evaluation is well grounded: the manuscript's only distributional justification for exchangeability is that both calibration and test descriptions are generated by the same model. The additional errors in Eq. (9) and Eqs. (5)-(6) mean the reported numbers do not even correspond to a well-defined conformal method. Given that the central contribution is an empirical trade-off claim, these issues are load-bearing and cannot be repaired by cosmetic revision. The paper may be resubmittable after reworking the method and adding human-validated experiments, but in its present form it does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper applies split conformal prediction to VLM-based place recognition and shows a trade-off curve between success rate and help rate on Matterport3D. The idea is sensible and the comparison against Prompt/Binary/No Help baselines is informative. But the evaluation never uses real user language, and the write-up contains internal contradictions (the no-fine-tuning claim vs. the fine-tuning step; a prediction-set definition that doesn't match the theorem). So the headline claims are not yet supported.\n\nWhat's new: applying CP to open-vocabulary room recognition with rank-based scores appears absent from the cited prior work, including KnowNo. The empirical finding that CP offers a continuous success/help trade-off where the baselines give one point each is plausible and would be useful if it holds.\n\nWhat's good: the problem framing is clear, the baselines are reasonable, and Matterport3D is appropriate. The authors correctly identify that conformal prediction gives distribution-free coverage under exchangeability.\n\nWhere it's soft:\n1. Exchangeability is asserted, not tested. Calibration and test descriptions are both produced by LLaVA from the same room images. The paper's motivating scenario is human ambiguity, but no human-written instruction appears anywhere in the evaluation. The coverage guarantee therefore holds only for LLaVA-style descriptions and gives no bound for real user language. This is the biggest gap.\n2. The 'no fine-tuning' claim contradicts the calibration steps, which describe fine-tuning the VLM before applying it. Either the method includes fine-tuning or the abstract should not claim otherwise.\n3. Equations (5)-(6) are hard to parse, and Eq. (9) adds an extra element to the prediction set beyond the standard threshold set. The coverage theorem in Eq. (4) applies to the threshold set, not to this modified set that the experiments actually use.\n4. No code or data are released, so the quantitative results can't be checked.\n\nNone of these are fatal to the underlying idea. With real user instructions, a corrected set definition, and a clean statement of whether anything is fine-tuned, this could become a solid applied contribution. As it stands, the evidence does not support the strong claims.\n\nWho it's for: people working on uncertainty-aware human-robot interaction or VLM-based navigation. A serious referee could help, but the paper needs substantial revision before acceptance. I'd send it to review with the expectation of major changes.","headline":"Reasonable CP-for-VLM-place-recognition idea, but the evaluation is self-referential and the write-up's contradictions undercut the guarantees.","tokens_in":7961,"tokens_out":4727,"would_cite":false,"duration_ms":45689,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Applying conformal prediction to a vision-language model's similarity scores gives indoor robots prediction sets with a statistical coverage guarantee while reducing how often they must ask for help.","keywords":["conformal prediction","uncertainty alignment","place recognition","vision-language models","assistive robotics","open-vocabulary classification","human-in-the-loop","built environments"],"falsifier":"Run SwPC on a set of Matterport3D rooms with human-written instructions that were not used in calibration, then measure the fraction of runs in which the true room is in the prediction set at the claimed $\\alpha$; if that empirical coverage drops below $1-\\alpha$ on a reasonably sized sample, the central coverage claim fails.","tokens_in":7107,"feed_emoji":"🤖","tokens_out":6961,"duration_ms":59416,"temperature":0.7,"pith_summary":"This paper proposes Seeing with Partial Certainty (SwPC), a wrapper that applies conformal prediction to the similarity scores a vision-language model assigns between a user's room description and top-down views of indoor spaces. The claim is that the resulting prediction set of candidate rooms comes with a statistical coverage guarantee: with probability at least $1-\\alpha$, the true room is in the set, for a user-chosen error rate $\\alpha$. On the Matterport3D indoor scene dataset, SwPC reaches higher success rates than prompting the LLM directly, and at matched success rates it asks for human help less often than the Prompt Set, Binary Set, and No Help baselines. Because the wrapper operates on the VLM's scores rather than its weights, it works with any VLM without fine-tuning. The significance is a lightweight, calibration-based way for assistive robots to know when they are uncertain and to ask for clarification instead of acting on a hallucinated answer.","feed_headline":"Room-recognition wrapper cuts robot help requests at same success rate","feed_subtitle":"Conformal prediction calibrates room candidates so robots ask for help only when truly unsure.","key_machinery":"The load-bearing object is the conformal prediction set built on the VLM's cosine similarity scores. For each room-image and description pair, the non-conformity score is $s = 1 - f(X,Y)$, where $f$ is the VLM similarity; the calibration set collects these scores over exchangeable pairs, and the quantile $\\hat q$ defined as in Eq. (8) sets the threshold. The prediction set $\\mathcal{C}(X_{\\mathrm{test}})$ is the smallest prefix of the VLM's ranked list whose scores fall at or below $\\hat q$. This mechanism is what transfers the conformal coverage guarantee of Eq. (4) to open-vocabulary place recognition, and it is what lets the robot decide, by set size larger than one, when to ask the user for clarification.","core_discovery":"The central discovery is that a conformal prediction layer over cosine similarity scores converts a VLM's raw ranking of rooms into a prediction set whose coverage is controlled. The paper constructs a calibration set by having a multimodal LLM generate room descriptions from top-down images, then recording non-conformity scores $s(X_i,Y_j)=1-f(X_i,Y_j)$ for the true and ranked candidates. Given a target error rate $\\alpha$, it takes the $\\lceil(n+1)(1-\\alpha)\\rceil/n$ quantile $\\hat q$ of the calibration scores and returns $\\mathcal{C}(X_{\\mathrm{test}})$ as the top-ranked rooms whose scores satisfy $1-f(X_{\\mathrm{test}},Y)\\le\\hat q$. The paper's experiments on Matterport3D show that this CP-based set achieves a success-versus-help trade-off curve dominating the LLM-prompted baselines: at the Prompt Set's average prediction-set size, CP's success rate exceeds the Prompt Set's, and at equivalent human-help levels CP gives higher success.","pith_inferences":["The evaluation generates its test instructions with the same multimodal LLM that produced the calibration descriptions, so the exchangeability assumption is satisfied by construction. A natural next test is to collect instructions from human users and measure whether the coverage guarantee survives that distribution shift; the paper's guarantee does not address that setting.","The calibration set is scene-specific (room descriptions paired with a fixed building's top-down views), so deploying in a new building likely requires a fresh calibration pass; an implicit extension is to pool calibration data across layouts or adapt the quantile online.","The same score-and-quantile recipe could be applied to other open-vocabulary detection tasks, such as object or affordance grounding, whenever a similarity score and a calibration set of paired inputs are available.","One could replace the fixed quantile with adaptive conformal inference to track drift in user language or environment over time, treating the paper's static calibration as a special case."],"forward_implications":["A robot can choose its operating point on the success-help trade-off by setting $\\alpha$, from always asking (set of size 1) to never asking (single best guess), without retraining the VLM.","The same calibration procedure can be dropped onto any open-vocabulary VLM classifier, so place recognition uncertainty alignment scales with improvements in foundational models.","The framework turns a hard safety question — 'when should the robot admit it does not know?' — into a quantile computation over a small calibration set.","At matched human-help rates on Matterport3D, SwPC achieves higher success than the Prompt Set, Binary Set, and No Help baselines, indicating that LLM-generated prediction sets were being biased or hallucinated.","Because the prediction set also serves as the clarification query (the candidates the robot lists), the human-in-the-loop interaction is determined directly by the calibrated uncertainty."],"supporting_citations":[{"why":"Supplies the conformal prediction theory and coverage guarantee used to construct prediction sets.","marker":"Angelopoulos and Bates 2023"},{"why":"Inspires the non-conformity scoring scheme and adaptive conformal inference under distribution shift.","marker":"Gibbs and Candès 2024"},{"why":"Provides the Matterport3D dataset on which all experiments are run.","marker":"Chang et al. 2017"},{"why":"LLaVA, the multimodal LLM used to generate room descriptions for calibration and test.","marker":"Liu et al. 2023"},{"why":"LongCLIP, the VLM backbone whose cosine similarity scores are calibrated.","marker":"Zhang et al. 2024"},{"why":"KnowNo, the prior uncertainty-alignment work for LLM planners that frames when a robot should ask for help.","marker":"Ren et al. 2023"},{"why":"Inner Monologue, source of the Binary Set baseline that prompts certainty/uncertainty.","marker":"Huang et al. 2022"},{"why":"Documents VLM/LLM hallucination, motivating the need for uncertainty alignment.","marker":"Farquhar et al. 2024"}],"fun_headline_variants":["Conformal prediction tells robots when to ask for help","Uncertainty layer keeps robot success up, help requests down","Robots ask for help only when truly lost","Calibrated room recognition slashes human intervention","Wrapper makes VLM place recognition honest about doubt"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The calibration descriptions and the instructions actually given to the robot must come from the same distribution, but the paper only tests LLaVA-generated descriptions, so the coverage guarantee has not been shown to hold for genuine human language.","fun_headline_variants_meta":{"raw":{"variants":["Conformal prediction tells robots when to ask for help","Uncertainty layer keeps robot success up, help requests down","Robots ask for help only when truly lost","Calibrated room recognition slashes human intervention","Wrapper makes VLM place recognition honest about doubt"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2528,"prompt_tokens":1001,"completion_tokens":1527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":1452}},"tokens_in":617,"tokens_out":1527,"duration_ms":11739,"temperature":1.0,"reasoning_tokens":1452,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:20:44.688406+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SwPC on a set of Matterport3D rooms with human-written instructions that were not used in calibration, then measure the fraction of runs in which the true room is in the prediction set at the claimed $\\alpha$; if that empirical coverage drops below $1-\\alpha$ on a reasonably sized sample, the central coverage claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the conformal prediction theory and coverage guarantee used to construct prediction sets."},{"cited_title":"and Cand\\` e s, E","cited_arxiv_id":null,"evidence_quote":"Inspires the non-conformity scoring scheme and adaptive conformal inference under distribution shift."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Matterport3D dataset on which all experiments are run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaVA, the multimodal LLM used to generate room descriptions for calibration and test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents VLM/LLM hallucination, motivating the need for uncertainty alignment."}],"review_version":1}