{"id":"42c97f25-e42a-47fc-b59f-a52f091bf35f","arxiv_id":"2508.08781","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ROOMELSA is a 44,445-query benchmark for mask-conditioned, language-guided 3D furniture retrieval, where top teams saturate top-10 recall but only reach 0.94 rank-1 accuracy on a 50-query private test.","lead":"This paper introduces ROOMELSA, a benchmark for retrieving the exact 3D furniture model a user points to in a cluttered room image while reading a short text description. The challenge results show coarse matching is largely solved, while fine-grained rank-1 ranking remains the open problem.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth IDs are human-chosen gallery proxies, not the exact scene meshes; retrieval scores may not reflect instance-level accuracy.","rationale":"The reader identified the human-selected gallery proxy as the weakest assumption, and the paper's text supports this. Since scenes are rendered from 3D-Future meshes, exact IDs are available; replacement to a 'more accurate match' introduces subjectivity and breaks the instance-level identity that the task claims. This is more fundamental than the small private test or author-participant overlap because it affects the validity of every ground-truth label and thus all reported scores. The test is feasible because the scene-mesh IDs are deterministically recoverable from the layout. If the IDs match, the concern is resolved; if not, the benchmark's interpretation must be revised to 'retrieval of annotator-preferred gallery models,' and the paper should quantify the discrepancy and justify why the proxy is the right target. Other issues (50-query test, author involvement) remain but are secondary. The paper has useful contributions—large-scale data, released dataset, detailed analysis—so conditional acceptance with a request for this analysis is appropriate.","tokens_in":17388,"tokens_out":7374,"duration_ms":67121,"concrete_test":"Take 200 random queries from the public split. From the 3D-FRONT layout files used in rendering, recover the exact 3D-Future mesh ID placed at the masked location. Compare this scene-exact ID with the published ground-truth gallery ID for the same query. Compute the agreement rate; if it is below 100%, re-run a top-performing method (e.g., the released pipeline of Stubborn_Strawberries) on the subset where the scene-exact mesh is present, using the scene-exact ID as ground truth, and compare MRR against the official scores. A material drop would show the proxy choice is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's ground-truth labels may not be the actual objects in the scenes. Section 4.2 Stage 2 says annotators 'replace the provisional CAD ID with a more accurate match selected from a gallery of 55,000 models.' Since every panorama is rendered from 3D-FRONT layouts using 3D-Future meshes, the exact mesh ID for each masked object is known a priori. The replacement step therefore implies the target is not the rendered instance but a human-selected gallery proxy. The triple-agreement protocol (IoU 0.87±0.04) measures consensus among raters, but not uniqueness or identity with the scene mesh. The paper does not report how often replacement occurred, nor whether the chosen proxy differs from the original layout mesh. Consequently, R@k and MRR measure retrieval to a proxy that annotators judged 'more accurate' for the mask/sentence, not necessarily the mesh visible in the scene. If multiple plausible gallery meshes exist for a query, the 'ground truth' is arbitrary, and conclusions like 'coarse retrieval solved' (R@5=R@10=1.00) and best MRR 0.97 are measured against that subjective choice. This directly threatens the paper's core claim of retrieving the 'exact CAD mesh' corresponding to the masked object.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ROOMELSA, a SHREC 2025 benchmark for mask-conditioned, language-driven 3D object retrieval. The task is: given a rendered panoramic room image, a binary mask highlighting an object, and a natural-language description, retrieve the correct CAD mesh from a gallery of 55,000 models. The dataset is built from 3D-FRONT/3D-Future assets, contains 44,445 verified (mask, sentence, mesh) triples over 1,622 apartments / 5,197 rooms, and is split into a public test (44,445 queries) and a private test (50 queries). The authors analyze the top five submissions, report that coarse retrieval is saturated (R@5=R@10=1.00 for all five), and identify fine-grained rank-1 ordering as the remaining bottleneck, with the best MRR at 0.97. They also describe the five winning pipelines in detail, emphasizing two-stage retrieval (fast vector search plus semantic reranking) as the dominant paradigm.","tokens_in":17662,"tokens_out":5167,"duration_ms":52864,"significance":"If the benchmark is sound, ROOMELSA fills a useful gap: existing 3D grounding datasets focus on localization (boxes or clusters), whereas ROOMELSA directly evaluates retrieval of a usable CAD asset from a gallery, conditioned on a mask and free-form language. The dataset is publicly released, the rendering pipeline (BlenderProc over 3D-FRONT/3D-Future) is reproducible, and the metric definitions are clear. The two-stage retrieval insight and the observation that rank-1 ordering, not coarse recall, is the bottleneck are likely to be valuable to the community. However, the empirical conclusions rest on a small, non-independent private test with no error bars, and the ground-truth annotation procedure allows replacement of the original scene mesh by a human-selected gallery proxy, which directly affects what the metrics actually measure. These issues substantially temper the strength of the claims as presented.","major_comments":[{"comment":"The ground-truth labels are not necessarily the actual objects in the scenes. Because every panorama is rendered from 3D-FRONT layouts with known 3D-Future mesh IDs, the 'provisional CAD ID' is the exact rendered instance. Yet Stage 2 says annotators 'replace the provisional CAD ID with a more accurate match selected from a gallery of 55,000 models.' This means the target of retrieval is a human-chosen gallery proxy, not the mesh visible in the scene. The reported inter-annotator IoU of 0.87±0.04 measures mask agreement, not uniqueness or identity of the selected mesh. The paper never reports how often replacement occurred, how far the proxy deviates from the rendered instance, or whether multiple plausible proxies exist. Since every metric (R@k, MRR) is computed against this label, the central claim of retrieving the 'exact CAD mesh' is not supported. Please either report replacement st","section":"§4.2 (Stage 2) and §4.1"},{"comment":"All five teams whose methods are analyzed in detail consist of co-authors of this paper (e.g., Stubborn_Strawberries includes Long Le Bao, Thai Hoang Minh et al.; Ai-Yahh includes Phu-Hoa Pham et al.; BUCCI_GANG, MealsRetrieval, and NoResources similarly overlap with the author list). This contradicts the statement in §5 that 'none of the ROOMELSA organizers participated in the challenge or submitted results.' The manuscript does not disclose this overlap. This is a significant transparency problem for a challenge report: the detailed methods, ablations, and qualitative comparisons are effectively self-authored evaluations. Furthermore, several method-specific constants (the sigmoid coefficients and threshold in Eq. (2), the mask-quality band in §6.3, and the voting weights in §6.5) appear tuned on the public test set, which is fully annotated and explicitly provided for local evaluation","section":"§5 and author list"},{"comment":"The private test consists of only 50 queries, and the authors correctly note that a single query counts as 0.02 in every metric. Yet the paper draws strong conclusions from small differences: R@1 values of 0.94, 0.92, 0.92, 0.90, and 0.88 correspond to 47, 46, 46, 45, and 44 correct rank-1 results. The abstract's claim that 'only one top-performing model consistently ranked the correct match first' is not supported without confidence intervals or significance tests (e.g., bootstrap or exact binomial pairwise tests). The paper's own analysis shows that the MRR spread between first and fifth place is three queries, yet it treats this as meaningful method-level ranking. Please provide error bars or significance tests, or limit the comparative claims to the well-supported statement that coarse retrieval is saturated and that rank-1 differences are small.","section":"§7.1 and Table 1"},{"comment":"The annotation description contains a duplicated sentence and an internal numerical contradiction. The text states: 'The model produces approximately 4,600 drafts per hour, providing complete coverage of all 44,000 masked objects ... The model produces approximately 4,600 drafts per hour, ensuring coverage of the more than 50,000 masked objects referenced in the Introduction.' The Introduction and the rest of the paper report 44,445 query triples; no 'more than 50,000 masked objects' is referenced anywhere. This is a factual inconsistency about the dataset size, which is a central contribution. Please fix the duplicate and clarify the exact number of masked objects covered by the automated drafts and the final verified set.","section":"§4.2 (Stage 1)"}],"minor_comments":[{"comment":"Remove the duplicated sentence in Stage 1; keep only one statement of throughput and coverage.","section":"§4.2"},{"comment":"'the team achieved the highest overall score in the challenge, establishing the ROOMELSA benchmark' is confusing; the benchmark is not established by the winning team. Rephrase.","section":"§6.1"},{"comment":"The private test is described as 'manually sample 50 diverse scenes, ensuring a balanced distribution across functional room types.' Please specify the exact room-type distribution and how diversity was ensured, as this affects generalizability.","section":"§4.3"},{"comment":"The typesetting of the equation has a missing line break or operator (wadapt appears to be multiplied by a bracket, but the expression is hard to parse). Please reformat.","section":"Eq. (2)"},{"comment":"'1024×512 pixels (internally specified with width = 512×2 and height = 512)' is an awkward way to state the resolution; simplify.","section":"§4.1"},{"comment":"The paper repeatedly says 'exact CAD mesh that corresponds to the described object' (e.g., §3.2, Fig. 2 caption). In light of the annotation replacement step, these statements should be revised to 'gallery mesh selected by annotators' unless the replacement issue is resolved.","section":"Global"}],"recommendation":"major_revision","confidential_remarks":"The author–participant overlap is the most serious concern for a SHREC/benchmark report. Even if the SHREC format expects the organizers to also be participants, this must be explicitly disclosed in the paper, and the leaderboard should be interpreted as an in-house evaluation rather than an independent competition. The editor should also check whether the journal has policies on self-authored challenge reports. The ground-truth proxy issue is fixable by additional analysis, but without it the benchmark's validity is questionable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about ROOMELSA up front. First, it's a genuinely new benchmark: given a panoramic 3D scene, a mask, and a natural-language description, the system retrieves a matching CAD mesh from a 55,000-model gallery. That reversed formulation—mask plus language instead of box prediction—is a real departure from ScanRefer and ReferIt3D, and the dataset itself (44,445 verified triples from 1,622 apartments) is a substantial resource. Second, the empirical evaluation has two flaws you need to weigh carefully. The ground-truth target is not necessarily the mesh that appears in the scene; Section 4.2 says annotators 'replace the provisional CAD ID with a more accurate match selected from a gallery of 55,000 models.' Since the scene is rendered from known 3D-Future meshes, the exact instance ID is available by construction. The replacement step means the target is a human-chosen proxy, and the paper never reports how often that happens or whether it changes the difficulty. That directly undermines the claim that top-10 recall is 'solved' against the exact scene objects.\n\nThe paper also has a serious conflict-of-interest problem: all five top teams listed in Section 5 are composed entirely of the paper's own co-authors. The text insists no organizers participated, but the author list and team lineups are nearly identical. That doesn't mean the methods are fake, but the leaderboard is not an independent evaluation, and the paper should say so plainly.\n\nWhat's good: the task definition is clean, the rendering and annotation pipeline are described in enough detail to reproduce, and the qualitative analysis of the two-stage 'retrieve then rerank' structure is useful. The observation that coarse recall saturates while rank-1 precision lags is likely correct—but on 50 private queries, with no error bars, the 0.03 MRR gap between first and fifth place is exactly three queries. That's too thin for the paper's confidence.\n\nMinor issues: the duplicate sentence in 4.2 and the 'more than 50,000 masked objects' vs. 44,445 discrepancy suggest the manuscript wasn't fully cleaned.\n\nBottom line: this is a valuable benchmark that deserves referee time, but the authors need to disclose the participant overlap, report the proxy-replacement statistics, and scale or caveat the private test. I'd take it in at C&G only with major revisions.","headline":"ROOMELSA is a genuinely new benchmark and a real resource, but its empirical claims are undercut by human-selected ground-truth proxies and an author-participant overlap that the paper doesn't disclose.","tokens_in":18331,"tokens_out":4121,"would_cite":true,"duration_ms":38327,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ROOMELSA establishes that coarse 3D object retrieval is solved while rank-1 fine-grained matching remains the open problem.","keywords":["3D object retrieval","benchmark dataset","mask-conditioned retrieval","language grounding","panoramic scene understanding","CAD gallery","fine-grained ranking","SHREC challenge"],"falsifier":"Take a random sample of ROOMELSA queries and have three independent annotators select the 'correct' gallery mesh for the same masked object. If they frequently disagree on the CAD ID, the ground-truth labels are non-unique, and reported MRR values partly reflect which proxy was chosen rather than retrieval skill. A second check: measure whether MRR degrades more than R@10 when masks are spatially shifted or sentences are paraphrased; if it does, the benchmark's remaining signal is largely tied to exact phrasing and mask placement.","tokens_in":17275,"feed_emoji":"🛋️","tokens_out":6550,"duration_ms":56603,"temperature":0.7,"pith_summary":"ROOMELSA is a new benchmark that asks a system to retrieve the exact 3D furniture mesh for a masked object in a panoramic room, given only a free-form text description. The dataset contains 44,445 verified (mask, sentence, 3D model) triples from 1,622 apartments and 5,197 rooms, rendered from 3D-FRONT layouts and 3D-FUTURE meshes. Across the 18 participating SHREC 2025 teams, the top five all achieve perfect Recall@5 and @10 on the 50-query private test, meaning the right mesh is always in the shortlist; differences in Mean Reciprocal Rank (best 0.97, fifth 0.93) come entirely from whether the correct mesh is ranked first. The paper argues this separates coarse retrieval, which is essentially solved, from fine-grained tie-breaking over materials, part structure, and contextual fit, which remains the open problem.","feed_headline":"New benchmark: top-10 retrieval solved, rank-1 remains the gap","feed_subtitle":"ROOMELSA's 44,445 mask-text queries show all top teams hit perfect recall@10 while MRR tops out at 0.97.","key_machinery":"The central object is the benchmark task itself: a query triple of an equirectangular panorama, a binary mask, and an attribute-rich sentence, with the answer being a ranked list from a gallery of roughly 55,000 CAD meshes. The carrying mechanism that emerges from the evaluation is the two-stage retrieval paradigm—a fast embedding-based vector search that guarantees top-k recall, followed by a semantic or geometric reranker that decides the final order. This shortlist-then-rerank structure is what lets near-perfect recall coexist with imperfect MRR.","core_discovery":"The paper establishes ROOMELSA as the first benchmark explicitly designed for mask-conditioned, language-driven 3D object retrieval, reversing the usual referring-object formulation: the location is given as a mask, and the system must decide which catalog CAD mesh matches the masked object's appearance and semantics. On the private test, coarse retrieval is saturated—every top-five method retrieves the correct mesh within the top ten for all 50 queries. The remaining signal is rank-1 ordering: the winning pipeline (multi-view SIGLIP embeddings, BLIP-2 captions, BGE-M3 similarity) reaches MRR 0.97, while a lightweight frozen-CLIP ensemble with silhouette embeddings and voting reaches 0.93. T","pith_inferences":["Because ground-truth CAD IDs are human-selected replacements from the gallery rather than the scene's original mesh, the benchmark's scores depend on annotator consistency; a natural extension is measuring inter-annotator agreement on CAD ID choice, which the paper does not report.","The saturation of R@10 at 1.00 across all top teams suggests the benchmark's discriminative power currently lives almost entirely in rank-1 ordering; shrinking the shortlist or adding near-duplicate distractors would make the evaluation harder and the leaderboard spreads wider.","The success of caption-based reranking implies that attribute-level or part-aware captions for gallery meshes might close the remaining gap more effectively than single-sentence captions.","A direct stress test would be to corrupt the mask or paraphrase queries and measure whether MRR drops faster than R@10, revealing how much of the tie-breaking depends on exact linguistic phrasing."],"forward_implications":["ROOMELSA separates the field into two measurable abilities: coarse recall, which is saturated at 1.00, and fine-grained rank-1 ordering, which decides leaderboard positions.","Improving MRR will require models that distinguish materials, part structures, and contextual/functional fit, not just category or silhouette.","A lightweight ensemble of frozen CLIP embeddings plus silhouette cues can reach MRR 0.93, so expensive multimodal training is not required for strong shortlisting.","The two-stage architecture (vector shortlist, then reranker) provides a concrete blueprint for deploying language-driven 3D retrieval in AR and robotics.","Enriching the gallery with material and part-level variants, or reducing the allowed top-k, would make the benchmark more discriminative for future rounds."],"supporting_citations":[{"why":"supplies the 3D-FRONT apartment layouts and 3D-FUTURE furniture meshes from which all ROOMELSA panoramas are rendered.","marker":"[3]"},{"why":"defines the prior ScanRefer task of 3D object localization from language, which ROOMELSA reverses by giving the mask and asking for the mesh.","marker":"[5]"},{"why":"provides the ReferIt3D benchmark with instance-level annotations that ROOMELSA extends from localization to gallery-level retrieval.","marker":"[4]"},{"why":"CLIP is the frozen vision-language backbone used by the lightweight ensemble and several other submitted pipelines.","marker":"[28]"},{"why":"SigLIP multi-view embeddings power the winning team's coarse vector search stage.","marker":"[36]"},{"why":"BLIP-2, fine-tuned on 3D renderings, generates captions for the winner's language-guided reranking.","marker":"[39]"}],"fun_headline_variants":["ROOMELSA: Top-10 retrieval solved, rank-1 remains the gap","Coarse 3D retrieval is solved, but rank-1 is still hard","New benchmark exposes the rank-1 gap in 3D retrieval","Mask-guided 3D retrieval: Recall high, exact match rare","Why finding the exact 3D object is still a challenge"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The ground truth for each query is a human-chosen CAD replacement from the 55,000-model gallery, and the benchmark's meaningfulness depends on that replacement being unique, consistent across annotators, and semantically identical to the masked object—properties the paper does not verify.","fun_headline_variants_meta":{"raw":{"variants":["ROOMELSA: Top-10 retrieval solved, rank-1 remains the gap","Coarse 3D retrieval is solved, but rank-1 is still hard","New benchmark exposes the rank-1 gap in 3D retrieval","Mask-guided 3D retrieval: Recall high, exact match rare","Why finding the exact 3D object is still a challenge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1260,"prompt_tokens":793,"completion_tokens":467,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":370}},"tokens_in":537,"tokens_out":467,"duration_ms":5254,"temperature":1.0,"reasoning_tokens":370,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:20:47.777851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of ROOMELSA queries and have three independent annotators select the 'correct' gallery mesh for the same masked object. If they frequently disagree on the CAD ID, the ground-truth labels are non-unique, and reported MRR values partly reflect which proxy was chosen rather than retrieval skill. A second check: measure whether MRR degrades more than R@10 when masks are spatially shifted or sentences are paraphrased; if it does, the benchmark's remaining signal is largely tied to exact phrasing and mask placement.","supporting_citations":[{"cited_title":"3d-future: 3d furniture shape with texture","cited_arxiv_id":null,"evidence_quote":"supplies the 3D-FRONT apartment layouts and 3D-FUTURE furniture meshes from which all ROOMELSA panoramas are rendered."},{"cited_title":"Scanrefer: 3d object localization in rgb-d scans using natural language","cited_arxiv_id":null,"evidence_quote":"defines the prior ScanRefer task of 3D object localization from language, which ROOMELSA reverses by giving the mask and asking for the mesh."},{"cited_title":"Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes","cited_arxiv_id":null,"evidence_quote":"provides the ReferIt3D benchmark with instance-level annotations that ROOMELSA extends from localization to gallery-level retrieval."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"CLIP is the frozen vision-language backbone used by the lightweight ensemble and several other submitted pipelines."},{"cited_title":"Sigmoid loss for lan- guage image pre-training","cited_arxiv_id":null,"evidence_quote":"SigLIP multi-view embeddings power the winning team's coarse vector search stage."},{"cited_title":"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models","cited_arxiv_id":null,"evidence_quote":"BLIP-2, fine-tuned on 3D renderings, generates captions for the winner's language-guided reranking."}],"review_version":1}