{"id":"8fd0c8b4-d851-4303-8fd9-85b9a7fbd59b","arxiv_id":"2506.14008","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new OOD object detection benchmark with near, far, and farther splits and open-set metrics shows close unknown objects are found more often but are also more frequently mistaken for known objects.","lead":"The paper introduces FindMeIfYouCan, a curated benchmark for out-of-distribution object detection that separates unknown objects into near, far, and farther semantic groups and adds open-set metrics that count ignored and misclassified unknown objects. It argues that the existing evaluation protocol hides in-distribution objects in its OOD images and cannot penalize detectors that simply ignore unknowns.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'farther' split changes both the ID dataset and the detector, so the claimed near/far/farther trends in Figure 5 may reflect detector/ID confounds rather than a single semantic-distance axis.","rationale":"The paper makes a well-documented case that the VOS benchmark violates its non-overlap assumption and that AUROC/FPR95 cannot penalize ignored objects; the OSOD metrics and the near/far splits for VOC are useful and reproducible contributions. However, the central empirical generalization — that localization difficulty and ID-confusion vary monotonically with semantic distance across near, far, and farther — depends on treating the BDD-based splits as a third level of the same distance scale. That treatment is not justified in Section 4.1: the farther splits differ simultaneously in the ID dataset, the detector training, the category space, and the image domain. Since BDD-trained detectors have markedly lower mAP (Table 3), the observed drops in APU/RU on farther splits could simply reflect weaker detectors rather than greater semantic distance. The paper's own CLIP similarity analysis (Figure 3) compares each ID to its own OOD sets, so it does not establish a common axis. This is the single most load-bearing concern because it undercuts the paper's headline finding, not merely a secondary detail. The reader's conditional verdict already points at this assumption; the proposed test would settle whether the trend survives when the detector and ID training are held fixed. If the trend reverses or flattens under a fixed detector, the benchmark is still useful as two independent settings (VOC-near/far and BDD-farther), but the three-level distance story should be retracted or reframed. I therefore keep the reader's CONDITIONAL verdict rather than moving to accept or reject.","tokens_in":34444,"tokens_out":4777,"duration_ms":52677,"concrete_test":"Run the VOC-trained Faster-RCNN (vanilla) on the COCO-farther and OpenImages-farther images, computing APU, RU, and nOSE against the same ground-truth labels, and compare with the published COCO-far/OpenImages-far values for the same detector. If RU does not decrease and nOSE does not decrease monotonically along near→far→farther for this fixed detector, the trends in Figure 5 are artifacts of switching from VOC-trained to BDD-trained models. Also report CLIP cosine similarity between Pascal-VOC and each of the six splits to check whether the 'farther' splits are actually farther from a common anchor than the VOC-far splits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 creates near/far splits only for ID=Pascal-VOC; for ID=BDD100k it only removes overlapping images and then labels the BDD OOD sets 'farther' based on Figure 3b and the results, without creating a third distance level relative to VOC. Consequently, every 'farther' data point in Figure 5 and Tables 14-25 comes from a detector trained on BDD, whereas near/far points come from detectors trained on VOC. Table 3 shows BDD-trained detectors are substantially weaker (mAP 31.2-33.3 vs 48.7-70.4), so the lower APU/RU and higher ignore rates on farther splits are confounded with detector quality. The headline conclusion, 'far and farther objects are harder to localize but less prone to be taken for an ID object,' requires that farther be a third level on the same semantic-distance scale, but the axis is never anchored to a common reference: the CLIP similarity comparisons in Figure 3 compare each ID dataset to its own OOD sets, not all OOD sets to a common distribution. Thus the central empirical trend is not established by the reported experiments as a function of semantic distance alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the standard VOS benchmark for out-of-distribution object detection (OOD-OD) violates its core non-overlap assumption because OOD images contain labeled and unlabeled in-distribution (ID) objects, and that the AUROC/FPR95 metrics built on that benchmark cannot measure whether unknown objects are actually localized, ignored, or confused with ID classes. To address this, the authors introduce the FMIYC benchmark, which removes overlapping images, creates near/far splits for Pascal-VOC and farther splits for BDD100k based on CLIP similarity and manual curation, and adds open-set metrics (APU, PU, RU, and the newly proposed nOSE). They evaluate several post-hoc OOD detection methods across Faster R-CNN, YOLOv8, and RT-DETR, and report that semantically near OOD objects are easier to localize but more often confounded with ID classes, whereas far and farther objects are harder to localize but less often mistaken for ID objects.","tokens_in":34672,"tokens_out":4873,"duration_ms":56364,"significance":"If validated, the paper would make a useful contribution: it documents a concrete overlap violation in an existing benchmark with counts and examples, releases a reproducible dataset and configuration files, and demonstrates that AUROC/FPR95 and open-set metrics are partially orthogonal (Figure 12). The breadth of architectures and methods evaluated is a strength, as is the concrete proposal of nOSE for cross-dataset comparability. However, the central empirical claim of a near/far/farther semantic-distance gradient is not established by the reported experiments, because the farther split changes both the ID dataset and the detector, and because the near/far ordering rests on manual curation without independent validation. The underlying idea is defensible, but the load-bearing comparison needs methodological repair before the headline conclusion can be accepted.","major_comments":[{"comment":"The 'farther' split is not a third level on the same semantic-distance axis as near and far. Near and far splits are defined only for ID=Pascal-VOC, while the farther splits are OOD sets with respect to BDD100k, and the detectors are trained separately on each ID dataset. Table 3 shows that BDD-trained detectors have substantially lower mAP (31.2–33.3) than VOC-trained detectors (48.7–70.4), so the lower APU/RU and higher ignore rates observed for farther splits in Figure 5 and Tables 14–25 are confounded with detector quality and domain shift. Figure 3 compares each ID dataset to its own OOD sets rather than anchoring all splits to a common distribution. To support the claim that far and farther objects are harder to localize but less confounded as semantic distance increases, the authors should either construct a third distance level within a single ID dataset with a fixed detector, or explicitly limit the trend claim to within-ID comparisons.","section":"§4.1, Figure 5, Table 3"},{"comment":"The near/far split is created by manually selecting near categories and then manually inspecting images, with no inter-annotator agreement, no random-selection control, and no independent validation of the ordering. CLIP similarity in Figure 3b is computed after the splits have been defined, so it reflects the manual category choices rather than providing an external test of the semantic-distance axis. Category-specific properties (object size, frequency, background) could drive the observed trends. The authors should add a per-category analysis or a random-split control to show that the measured effects are attributable to the semantic-distance gradient rather than to the particular categories chosen for each split.","section":"§4.1, Table 6, Figure 3b"},{"comment":"The definition of the OSOD metrics in a post-hoc OOD-OD setup is incomplete for predictions that do not overlap any ground-truth unknown object. The base detector emits ID-class labels; a detection labeled as an ID class on an OOD image that does not overlap a labeled unknown is neither a TP_U, an FP_U, nor an F_NM under the stated definitions, so it is not counted by PU, RU, or nOSE. Since the benchmark's OOD images are constructed to contain no ID objects, such spurious ID-class detections are exactly the failure mode the benchmark is meant to expose. The authors should specify how such predictions are treated, or add a metric that captures them; otherwise the claim that OSOD metrics 'comprehensively' evaluate confusion with ID objects is overstated.","section":"§4.2 and Appendix D, Eq. (3)–(5), Eq. (9)"}],"minor_comments":[{"comment":"The sentence describing nOSE says it measures 'more objects are mistakenly predicted as one of the in-distribution (ID) classes among the correctly localized objects,' but the definition in Eq. (9) normalizes by all ground-truth unknowns (TP_U + FN_U), not only by correctly localized ones. Please correct the text to match the equation.","section":"§5.3, text near Figure 5"},{"comment":"Several panels in Figure 5 use a broken or non-linear y-axis (e.g., ticks at 0, 10, 21.8, 30, 100), which can visually exaggerate differences. Use a consistent linear scale.","section":"Figure 5"},{"comment":"The terms near, far, and farther are typeset inconsistently (italic, roman, and a mix in the abstract). Please unify the formatting for readability.","section":"Abstract and Section 4.1"},{"comment":"The reference to Du et al. 2022b is rendered as 'V os' rather than 'VOS' in the bibliography; correct the typo.","section":"References"},{"comment":"Table 4 reports the number of OOD images containing ID classes, but it would be helpful to also report the total number of images inspected and the inter-annotator agreement for the manual verification, since the curation is a key contribution.","section":"Appendix B, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a benchmark contribution with solid overlap documentation and reproducible artifacts, but the central claim about a continuous near/far/farther semantic-distance trend is currently confounded by the change of ID dataset and detector. The fix is feasible within the paper's scope: construct a true third distance level under a single ID dataset or restrict the trend claim to within-ID comparisons. If the authors cannot do that, they should reframe the contribution as the overlap fix and the metric integration, and soften the cross-split trend conclusions. The manual curation without inter-rater reliability is also a concern for a dataset paper and should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper's central criticism of VOS is correct and documented: the OOD splits contain labeled and unlabeled ID objects, so AUROC/FPR95 on those splits are built partly on predictions that are actually correct localizations of ID classes, and they cannot see ignored unknown objects at all. That is a genuine problem for the only widely used OOD-OD benchmark. Second, the proposed FMIYC splits plus OSOD metrics (APU, PU, RU, normalized open-set error) are a sensible extension, and the experiments are broad: three detector families, many post-hoc OOD scoring methods, two ID datasets.\n\nWhat is actually new is the semantic stratification by CLIP similarity and the use of OSOD metrics to measure whether unknowns are ignored or confused with ID objects. The observation that near OOD objects are easier to localize but more confounded is interesting and runs opposite to image classification intuitions.\n\nSoft spots, in order of severity. The farther split is not actually a third level on a single semantic-distance axis. It changes the ID dataset from VOC to BDD, which also changes the detector training data and mAP (roughly 31-33 vs 49-70). So the Figure 5 trend \"near/far/farther\" is partly a detector/dataset confound. The CLIP similarities in Figure 3 compare each ID dataset only to its own OOD sets, so they do not anchor VOC and BDD on a common scale. This does not kill the benchmark—the near/far comparison within VOC is fine—but it does mean the headline claim about farther is not established. Second, the near/far curation is manual, with no inter-annotator agreement or random-selection baseline; the authors do provide image-ID config files and code, which is more than most benchmark papers do, so this is a reproducibility plus but a validity concern. Third, Section 6 claims no work has proposed a specific VLM method for OOD-OD while citing RUNAL (Zhang et al. 2025), which is exactly an object-level OOD method using multimodal representations. That overclaim should be corrected.\n\nWho this is for: anyone working on OOD-OD or open-set detection, and anyone using VOS as a benchmark. It deserves referee time. The fixes are substantial but bounded: label the farther condition as BDD-anchored and stop treating it as the same distance axis, or rebuild it with a common ID reference; add curation reliability checks; fix the VLM claim.","headline":"Useful benchmark and a real critique of VOS, but the near/far/farther trend rests on a confounded comparison; worth refereeing with fixes.","tokens_in":35219,"tokens_out":2204,"would_cite":true,"duration_ms":22690,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The standard OOD object-detection benchmark hides ignored objects; new near/far/farther splits expose that mistake.","keywords":["out-of-distribution object detection","open-set object detection","benchmark curation","semantic distance","near-far-farther splits","unknown recall","unknown precision","normalized open-set error"],"falsifier":"Construct a single-ID benchmark using only Pascal-VOC as in-distribution data, with three manually separated OOD levels whose measured CLIP-distance gaps match the near-far and far-farther gaps in FMIYC, and rerun the same methods; if the nOSE and recall trends do not reproduce a three-step gradient, the 'farther' label was not doing the work.","tokens_in":34241,"feed_emoji":"🎯","tokens_out":3943,"duration_ms":38616,"temperature":0.7,"pith_summary":"The paper argues that the standard benchmark for out-of-distribution object detection (OOD-OD) silently violates its own no-overlap rule: its OOD images contain in-distribution objects such as people and dining tables, so every prediction on those images is treated as wrong even when the detection is correct. Because the benchmark's AUROC and FPR95 scores are computed only from predictions, they cannot penalize a detector for ignoring unknown objects entirely. To fix this, the authors curate the FindMeIfYouCan (FMIYC) benchmark, removing overlaps and splitting OOD data by semantic distance into near, far, and farther levels, and add open-set metrics (APU, PU, RU, nOSE) that use ground-truth boxes. Across Faster R-CNN, YOLOv8, and RT-DETR, the new metrics show semantically close OOD objects are easier to localize but also the most often mistaken for ID objects, while far and farther objects are rarely localized but, when detected, are less confounded with ID classes.","feed_headline":"New benchmark shows detectors miss far objects but mistake near ones","feed_subtitle":"Curated near/far/farther OOD splits with open-set metrics reveal when unknown objects are ignored or confused with known classes.","key_machinery":"The load-bearing object is the curated evaluation split with ground-truth labels: distance categories near, far, and farther determined by CLIP cosine similarity plus manual inspection, together with the OSOD metrics APU, PU, RU, and nOSE computed at an IoU threshold of 0.5. The mechanism is that ground-truth boxes let the benchmark count ignored unknowns through low recall and confusions through nOSE, instead of only scoring the detector's confident predictions.","core_discovery":"The central claim is that the widely used VOS benchmark for OOD object detection cannot measure whether unknown objects are found, ignored, or confused with known classes. The authors show that OOD images in that benchmark contain labeled and unlabeled instances of ID classes, such as people and dining tables, violating the non-overlap assumption, and that up to roughly 59% of images in one OOD split produce no predictions above threshold, so AUROC and FPR95 are computed on a biased subset. The proposed FMIYC benchmark removes overlaps, adds curated near and far splits relative to Pascal-VOC, and designates the BDD-based OOD sets as farther; it then applies open-set metrics that use ground-truth boxes to quantify unknown recall, precision, and the normalized open-set error. The result is that near OOD objects are localized better but also suffer the highest nOSE, while farther objects have very low recall and very low nOSE, a pattern invisible to the old metrics.","pith_inferences":["The near/far/farther ordering is not a single validated distance axis because the farther level is defined relative to BDD100k rather than Pascal-VOC; a testable extension would build a true three-level split within one ID dataset.","Because the benchmark depends on exhaustive ground-truth labels, unlabeled OOD objects that are detected will be counted as false positives; the paper acknowledges this and keeps AUROC/FPR95 for that case, so a hybrid reading is needed.","The finding that near objects are easier to localize than far ones inverts the usual image-classification intuition; a natural next experiment is to ablate localization difficulty separately from semantic similarity to see which factor drives the trend."],"forward_implications":["If the protocol is adopted, prior OOD-OD results reported on the VOS benchmark need re-examination because their AUROC and FPR95 numbers are built on a contaminated and biased subset.","Detector architectures and scoring functions can now be compared on whether they actually find unknowns, not just whether they flag incorrect predictions; for example, density-based scores excel on farther splits while others lead on near splits.","Tuning the detection threshold becomes an explicit lever: lowering it gives scoring functions more candidate boxes and could raise unknown recall.","The correlation analysis shows that unknown precision is orthogonal to AUROC and FPR95, so claims about OOD-OD safety should report open-set metrics as well.","The benchmark enables a direct comparison of open-set metrics with legacy discrimination metrics on the same architecture, method, and split, which was previously impossible."],"supporting_citations":[{"why":"Supplies the original VOS benchmark and model checkpoints whose non-overlap assumption and AUROC/FPR95 protocol are the paper's main critical target.","marker":"Du et al. [2022b]"},{"why":"Formalizes open-set object detection and introduces the AOSE, wilderness impact, and related metrics that FMIYC imports and adapts.","marker":"Dhamija et al. [2020]"},{"why":"Provides the CLIP-cosine-similarity procedure the paper uses to justify the near/far/farther split.","marker":"Mayilvahanan et al. [2023]"},{"why":"Supplies the CLIP image encoder whose embedding space is used to measure semantic and visual similarity between ID and OOD datasets.","marker":"Radford et al. [2021]"},{"why":"Provides the unified OSOD problem formulation and metric conventions used for APU, PU, RU, and nOSE.","marker":"Ammar et al. [2024]"},{"why":"Defines Pascal-VOC, the in-distribution dataset for the near and far splits.","marker":"Everingham et al. [2010]"},{"why":"Defines BDD100k, the in-distribution dataset whose OOD sets are labeled farther.","marker":"Yu et al. [2020]"},{"why":"Provides the COCO images and annotations that are curated into the COCO-near and COCO-far splits.","marker":"Lin et al. [2014]"},{"why":"Provides the OpenImages images and annotations that are curated into the OpenImages-near and OpenImages-far splits.","marker":"Kuznetsova et al. [2020]"}],"fun_headline_variants":["Near OOD objects confuse detectors, far ones are missed","Open set metrics: near unknowns misclassified, far unknowns ignored","FMIYC benchmark: near OOD misclassified, far OOD missed","Detectors misclassify near unknowns, miss far ones","Benchmark fix: near unknowns confound, far unknowns invisible"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire near/far/farther narrative assumes that CLIP similarity plus manual curation across two different in-distribution datasets, Pascal-VOC and BDD100k, yields one meaningful distance axis; if 'farther' just means 'BDD's OOD images,' the observed trend is not purely a distance effect.","fun_headline_variants_meta":{"raw":{"variants":["Near OOD objects confuse detectors, far ones are missed","Open set metrics: near unknowns misclassified, far unknowns ignored","FMIYC benchmark: near OOD misclassified, far OOD missed","Detectors misclassify near unknowns, miss far ones","Benchmark fix: near unknowns confound, far unknowns invisible"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001356,"raw_usage":{"total_tokens":5548,"prompt_tokens":1032,"completion_tokens":4516,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":4429}},"tokens_in":648,"tokens_out":4516,"duration_ms":29806,"temperature":1.0,"reasoning_tokens":4429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:25:03.370535+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a single-ID benchmark using only Pascal-VOC as in-distribution data, with three manually separated OOD levels whose measured CLIP-distance gaps match the near-far and far-farther gaps in FMIYC, and rerun the same methods; if the nOSE and recall trends do not reproduce a three-step gradient, the 'farther' label was not doing the work.","supporting_citations":[],"review_version":1}