{"id":"b110ea40-98e0-49e7-9fcd-9bc21017eec9","arxiv_id":"2608.06850","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RegionDet shows supervised detectors can partially learn to localize activity and context regions, while open-vocabulary detectors nearly fail, indicating current vision-language models are object-centric.","lead":"The paper introduces RegionDet, a COCO-style benchmark with eight region categories such as construction areas, queues, and talking groups, annotated with bounding boxes. It evaluates 29 detector variants and finds supervised models reach about 37 AP while zero-shot vision-language models stay near 1 AP, pointing to a strong object-centric bias.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No inter-annotator agreement is reported for boxes around boundary-less region targets; all AP claims in Tables 4-5 inherit this unresolved measurement-validity risk.","rationale":"I considered two candidate concerns. First, the zero-shot results might be sensitive to the fixed category-name prompts, since open-vocabulary detectors are typically aligned to noun phrases rather than state or activity words. That concern is real but less load-bearing: the gap is enormous, and the qualitative failure mode (predictions attached to individual objects rather than region extents) already supports the object-centric-bias interpretation, so a prompt re-run would be informative but unlikely to overturn the main conclusion. Second, and more fundamentally, the benchmark's ground-truth boxes for amorphous region targets may not be reproducible. The paper explicitly describes instructions for annotators to choose 'the spatial extent that best covers the intended semantic, functional, or activity region,' but provides no inter-annotator agreement statistic. Since both the supervised training targets and the evaluation boxes come from the same subjective process, any annotation inconsistency propagates directly into every AP value in Tables 4 and 5. This is exactly the reader's weakest_assumption, and I agree with it. The reader's conditional verdict is appropriate: the concern is concrete, testable, and addressable before public release, but it does not justify rejection because the reported two-round verification process may well have produced acceptable consistency. No change to the verdict is needed.","tokens_in":12815,"tokens_out":5680,"duration_ms":55729,"concrete_test":"Have a second independent annotation team re-annotate a stratified random sample of at least 100 test images per category using the same written guidelines. Compute (a) per-category mean and median IoU between original and re-annotated boxes, and (b) AP of a fixed detector (e.g., RF-DETR-L) evaluated against both annotation sets. If per-category mean IoU is below about 0.5, or if any category's AP shifts by more than about 2 points, the numbers in Tables 4 and 5 are not stable enough to support the paper's conclusions without a revised annotation protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—closed-set detectors reach about 37 AP while zero-shot detectors stay near 1 AP—rests entirely on the reproducibility of RegionDet's ground-truth boxes. The 'Data Collection and Annotation' section instructs annotators to 'annotate the spatial extent that best covers the intended semantic, functional, or activity region' for categories such as Talking, Queuing, Waiting, and Damage. These targets lack stable physical boundaries: a group conversation can be boxed tightly around two people or more broadly around the interaction zone, and a waiting area can be drawn around individuals or the functional space. The paper reports two rounds of verification but no inter-annotator agreement, no per-category box IoU distribution, and no annotation retest study. Every AP number in Tables 4 and 5 is therefore a joint measure of detector error and annotation-convention noise. The size of the supervised-vs-zero-shot gap (37 vs 1 AP) makes it likely that the qualitative conclusion survives, but the precise AP ranking and the claim that closed-set detectors 'partially learn region-level patterns' are not yet quantitatively grounded. This is a measurement-validity assumption, distinct from the model-capability claim, and it is the load-bearing point on which the benchmark's usefulness rests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Region Detection, a detection task whose targets are not discrete object instances but regions defined by states, relations, scene functions, and context (e.g., Construction, Damage, Queuing, Talking). It constructs RegionDet, a dataset of 8,010 images with 12,588 COCO-style bounding-box annotations across eight categories, and evaluates representative closed-set and zero-shot/open-vocabulary detectors. The main empirical findings are that supervised detectors partially learn these regions, reaching up to 37.6 AP, while text-conditioned detectors essentially fail, with AP near 1.0, which the authors attribute to the object-centric bias of current vision-language detectors.","tokens_in":13039,"tokens_out":4704,"duration_ms":44433,"significance":"If the annotations are reliable, RegionDet addresses a genuine gap in detection evaluation: most benchmarks focus on well-bounded object instances, and the dataset provides a concrete testbed for state-, relation-, and context-defined localization. The paper's strengths are its broad coverage of detection paradigms, direct measurement of external detectors, standard COCO-style evaluation, and the absence of any circularity in the central comparison. The 37-vs-1 AP gap is a striking and potentially useful result. However, the significance is conditional on the validity and reproducibility of the ground truth itself, because most region categories lack stable physical boundaries and no inter-annotator agreement is reported.","major_comments":[{"comment":"The paper reports two rounds of verification but gives no inter-annotator agreement statistics. This matters directly for every AP number in Tables 4 and 5: for categories such as Talking, Queuing, Waiting, and Damage, annotators were instructed to 'annotate the spatial extent that best covers the intended semantic, functional, or activity region,' and these extents are subjective. A round of 'verification' and discussion of disagreements does not quantify the resulting box variability. Without pairwise IoU or box-level agreement values, reported AP scores jointly measure detector error and annotation-convention noise, and the claim that closed-set detectors 'partially learn region-level patterns' is not cleanly grounded. I ask for an annotation agreement study (e.g., per-category IoU between independent annotators, or on a held-out set), together with the annotation guideline and a description of how ambiguous cases were resolved.","section":"Data Collection and Annotation"},{"comment":"No error bars, multiple seeds, or significance tests are reported. The top two methods, RF-DETR-L (37.6 AP) and DEIMv2-L (37.0 AP), differ by 0.6 AP, which is within typical run-to-run variance for detection training; RT-DETRv4-L is at 33.1 AP. The text's statement that 'RF-DETR-L achieves the best overall AP' and the broader conclusion that query-based Transformer detectors are more effective are therefore not robust as stated. I ask the authors to report mean and standard deviation over at least three training seeds, or to rephrase the ranking claims accordingly.","section":"Closed-set Detection Results, Table 4"},{"comment":"The manuscript states that 'The RegionDet will be released' but provides no dataset URL, license, or concrete release plan, and no annotation statistics beyond per-category counts (e.g., box size distribution, boxes-per-image distribution, or overlap between region boxes). Because the dataset is the central contribution, the absence of a public release mechanism and of descriptive annotation statistics limits independent verification of all experiments and the interpretability of the AP values. I ask the authors to include the release link (or at least an availability statement with a timeline), a dataset card, and basic annotation statistics in the final version.","section":"Abstract / Dataset Statistics"}],"minor_comments":[{"comment":"The YOLOv13-L row reports '6.61' for Damage, which is written with two decimals while all other entries in the table use one decimal; this appears to be a typo and should be corrected or unified.","section":"Table 4"},{"comment":"The sentence 'Following the evaluation protocol in Section Evaluation Protocol' contains a dangling self-reference; it should point to the actual section number (e.g., Section 3.4).","section":"Experimental Setup"},{"comment":"The sentence 'The training and test sets are approximately balanced within each category' is ambiguous; it means that each category has a roughly 50/50 train/test split, but the overall class frequencies are highly imbalanced (e.g., Damage has 4,739 regions while Queuing has 426). Clarifying this would prevent confusion.","section":"Dataset Statistics"},{"comment":"The annotation guidelines do not discuss how to handle images where multiple region categories overlap (for example, a damaged area inside a construction site or a group that is both queuing and talking); a short statement on exclusive versus coexisting region annotations would be helpful.","section":"Data Collection and Annotation"},{"comment":"Figures 3 and 4 show selected qualitative examples but do not specify the score threshold or IoU threshold used for visualization; adding this information would make the qualitative results more reproducible.","section":"Qualitative Analysis"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution—a box-level benchmark for state-, relation-, and context-defined regions, with eight categories and 29 detectors evaluated under a clean COCO-style protocol. The headline finding, that supervised detectors get to roughly 37 AP while zero-shot vision-language detectors hover near 1 AP, is almost certainly robust; the gap is too large to be measurement noise. Credit where due: the task definition is sensible, the categories cover a useful slice of urban and surveillance scenarios, and the paper honestly reports per-category results and shows qualitative examples that support the numbers.\n\nThe soft spot is the one the stress-test note hits. The ground truth is boxes around targets that do not have physical boundaries—Talking, Queuing, Waiting—and the annotation instructions are necessarily underdetermined. The paper says ten annotators drew boxes and five did two verification rounds, but there is no inter-annotator agreement, no box-IoU distribution, no retest statistic. Every AP in Tables 4 and 5 is a joint measure of detector error and annotation-convention noise. The 37-vs-1 gap is robust to that noise, but the precise rankings among closed-set detectors, and the claim that closed-set detectors can partially learn region-level patterns, are not quantitatively grounded until annotation consistency is documented. Also missing: run variance or multiple seeds for the detector evaluations, so the top differences (RF-DETR-L at 37.6 vs DEIMv2-L at 37.0) are within likely noise. The dataset is not yet released, which limits reproducibility.\n\nNone of this is fatal. The qualitative conclusions are well supported by the examples, and the zero-shot failure is consistent across four families and many backbones. The paper would benefit from an annotation-consistency study, error bars, and a release of the data. That is a standard referee request.\n\nWho should read this: anyone working on open-vocabulary detection, context-aware localization, or urban scene understanding. It deserves serious peer review; a competent reviewer can push on the annotation quality and the paper will come back stronger.","headline":"A genuinely useful new benchmark for region-level detection, with a likely real 37-vs-1 AP gap, but the missing inter-annotator consistency numbers keep it from being fully trustworthy.","tokens_in":13572,"tokens_out":2475,"would_cite":true,"duration_ms":24055,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark tests whether detectors can find regions, not just objects, and current zero-shot models fail badly.","keywords":["region detection","benchmark","object detection","open-vocabulary detection","object-centric bias","bounding boxes"],"falsifier":"Measure inter-annotator agreement on a sample of RegionDet images, and check whether supervised detector AP scores change significantly when the box ground truth is re-annotated by a different set of raters.","tokens_in":12625,"feed_emoji":"📦","tokens_out":781,"duration_ms":8673,"temperature":0.7,"pith_summary":"This paper introduces Region Detection, a task that extends object detection from discrete, well-bounded instances to visual regions defined by states, relations, scene context, and activities. It builds RegionDet, a benchmark with eight such categories, and evaluates both supervised and zero-shot detectors. The central claim is that closed-set detectors can partially learn these region-level patterns, while current vision-language detectors fail severely, exposing a strong object-centric bias.","feed_headline":"New benchmark shows detectors fail on regions, not just objects","feed_subtitle":"Closed-set models reach 37.6 AP on region targets; zero-shot models drop to 1.0 AP.","key_machinery":"The central object is the RegionDet benchmark itself: 8,010 images with 12,588 COCO-style bounding-box annotations over eight region categories. The key mechanism is the contrast between supervised closed-set training and zero-shot text-query evaluation, which isolates whether detectors can learn region-level patterns without explicit object supervision.","core_discovery":"The paper establishes that region targets defined by states, relations, and context are learnable under supervision but remain hard, and that zero-shot/open-vocabulary detectors essentially cannot localize them. On the test set, the best supervised detector (RF-DETR-L) reaches 37.6 AP, while the best zero-shot detector reaches only about 1.0 AP. The ability to localize regions such as Queuing, Talking, and Damage falls sharply compared to object-centric categories, indicating that existing detectors rely on object appearance rather than relational and contextual reasoning.","pith_inferences":["If zero-shot detectors could be fine-tuned on region-level data, their relative weaknesses across categories could guide where to add relational reasoning modules."],"forward_implications":["Closed-set detectors, especially query-based Transformers, can serve as a baseline for future region-detection research."],"supporting_citations":[{"why":"MS COCO defines the box-annotation format and evaluation protocol that RegionDet adopts.","marker":"Lin et al. 2014"},{"why":"Grounding DINO provides the text-conditioned detection architecture evaluated zero-shot.","marker":"Liu et al. 2024"},{"why":"YOLO-World supplies a real-time open-vocabulary detector baseline.","marker":"Cheng et al. 2024"},{"why":"YOLOE evaluates open-vocabulary detection with object-like queries and serves as the qualitative failure case.","marker":"Wang et al. 2025"},{"why":"RF-DETR-L provides the strongest supervised baseline in the closed-set comparison.","marker":"Robinson et al. 2025"}],"fun_headline_variants":["RegionDet: zero-shot detectors drop to 1 AP on region targets","Supervised detectors learn regions, zero-shot miss them entirely","New benchmark shows object-centric detectors fail at regions","Region detection: closed-set learns, open-vocab fails hard","Beyond object instances: benchmark exposes context blind spot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth boxes for region targets such as Talking and Waiting are annotated by human raters, and the paper relies on those boxes being consistent even though the regions lack stable physical boundaries and no inter-annotator agreement is reported.","fun_headline_variants_meta":{"raw":{"variants":["RegionDet: zero-shot detectors drop to 1 AP on region targets","Supervised detectors learn regions, zero-shot miss them entirely","New benchmark shows object-centric detectors fail at regions","Region detection: closed-set learns, open-vocab fails hard","Beyond object instances: benchmark exposes context blind spot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1501,"prompt_tokens":880,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":539}},"tokens_in":496,"tokens_out":621,"duration_ms":6405,"temperature":1.0,"reasoning_tokens":539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:37:24.363210+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure inter-annotator agreement on a sample of RegionDet images, and check whether supervised detector AP scores change significantly when the box ground truth is re-annotated by a different set of raters.","supporting_citations":[],"review_version":1}