{"id":"71787589-cb4b-496e-8c1a-415db7464c5a","arxiv_id":"2511.20274","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An auto-generated 615K-image dataset with action, object, and relation annotations and a tri-level CLIP variant show in-distribution gains over PyramidCLIP, without external validation.","lead":"This preprint introduces SCLARO/Action-Genome, a 615,805-image dataset with action captions, object boxes, and relation triplets generated automatically by vision-language models, plus ScenarioCLIP, a CLIP-style model with separate global/object/relation encoders. It reports gains over PyramidCLIP in retrieval and detection, but all evaluation uses the same auto-generated labels as ground truth.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All benchmark numbers are measured against auto-generated VLM/GroundingDINO/SAM labels with no human or external verification; if those labels are noisy, the 'grounded' dataset claim and every reported gain lose their meaning.","rationale":"The reader's weakest assumption identifies the exact load-bearing concern: annotations are generated entirely by a VLM plus grounding/segmentation models, with no human verification or external agreement check, and all evaluation is measured against those same labels. If the auto-generated objects, relations, and boxes are noisy or hallucinated, then the 'grounded' dataset claim collapses and the reported ScenarioCLIP gains may reflect fitting the generator's quirks rather than real scene understanding. The paper does describe a plausible pipeline, and the architecture/ablation details are sufficient for re-implementation, but none of that substitutes for label-quality evidence. The missing OOD experiments promised in the abstract further weaken the generalization claim, though the annotation-quality issue is primary. Since the reader already assigned CONDITIONAL, my read does not change that verdict: a human audit and a human-confirmed evaluation subset would either validate the dataset or require rejection/withdrawal of the central claims.","tokens_in":18923,"tokens_out":3530,"duration_ms":39723,"concrete_test":"Sample 500 images per constituent source (OpenPSG, PVSG, Kinetics, MMT, SemanticKITTI), stratified by image and annotation length. Have at least two independent annotators mark: (1) action caption correctness, (2) object bbox validity / IoU against manual boxes, (3) relation triplet correctness and whether the focused region actually shows that relation. Compute precision and inter-annotator agreement separately per level. Then rerun zero-shot retrieval (Table 1) and object detection (Table 3) only on the human-confirmed test subset; if ScenarioCLIP's gains over PyramidCLIP shrink or reverse, the headline improvements are artifacts of auto-label noise. A pass threshold could be, e.g., ≥80% relation precision and ≥0.5 mean box IoU.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central 'grounded' claim depends on the correctness of the annotation pipeline, but the paper provides no measurement of it. Sections 3.1/3.2 and §7 describe a fully automatic generation: Ovis-Gemma 9B produces action captions, object lists, and relation triplets; GroundingDINO boxes objects; SAM segments masks; and RBF blending creates 'focused regions.' The relation itself is never independently grounded: the focused region is just a spatial blend of the two object masks, not verified to correspond to the relation. Crucially, every evaluation in Tables 1/2/3/5/6 is performed against these same auto-generated labels, with no human audit or external agreement check. The dataset therefore measures consistency with the generator, not scene understanding. For example, if Ovis hallucinates a relation triplet, ScenarioCLIP is rewarded for retrieving that hallucinated label, and object detection rewards boxes that may only exist as VLM artifacts. This does not make the internal comparison meaningless, but it removes the basis for the resource claim and makes reported gains untrustworthy as evidence of grounded scenario understanding. The only possible external validation—reusing action classes from Kinetics/MMT and relations from OpenPVSG—is mentioned in one sentence in §3.2, but no numbers, splits, or quality analysis are given. The abstract also promises out-of-domain generalization, but no OOD experiments appear in the main text or supplementary.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces a large-scale visual scene understanding resource (615,805 images) with three levels of annotation: global action captions, object bounding boxes, and relation triplets, together with relation-focused regions. The annotations are produced entirely automatically by a pipeline comprising a VLM (Ovis-Gemma 9B), GroundingDINO, and SAM, with only vocabulary-level curation. The paper also proposes ScenarioCLIP, a CLIP-style model with disentangled global, object, and relation encoders trained with contrastive alignment and EMA-based knowledge distillation. Experiments on the proposed dataset report gains over PyramidCLIP on zero-shot retrieval, linear probing, object detection, predicate classification, and scene graph classification, and the abstract claims out-of-domain generalization. The central claims are (a) the dataset provides 'grounded' scenario-level annotations and (b) ScenarioCLIP improves relational scene understanding relative to prior CLIP-style models.","tokens_in":19326,"tokens_out":3263,"duration_ms":36087,"significance":"If the dataset annotations were trustworthy, the resource would be potentially valuable: it is large, spans diverse domains, and combines action, object, and relation supervision in one corpus, which is uncommon. The ScenarioCLIP architecture is also a reasonable extension of CLIP-style pretraining to hierarchical scene understanding, and the disentangled-encoder design with EMA distillation is clearly described. The paper ships code and a data pipeline, which aids reproducibility. However, the significance is conditional on the correctness of the annotations and on the validity of the evaluation. As it stands, the evaluation is self-referential: all labels used for training and testing are generated by the same automatic pipeline, with no human verification or external benchmark. This undermines the dataset's 'grounded' claim and makes the reported performance gains hard to interpret as evidence of real scene understanding rather than of agreement with a particular VLM's biases.","major_comments":[{"comment":"The evaluation is circular. All ground-truth action, object, and relation labels are produced automatically by Ovis-Gemma, GroundingDINO, and SAM, with no human verification or external agreement check. Every number in Tables 1, 2, 3, 5, and 6 is measured against these auto-generated labels. If the VLM hallucinates objects or relations, or if GroundingDINO/SAM mislocalize them, the reported gains may simply reflect how well ScenarioCLIP fits the generator's output. The manuscript should include (i) a human evaluation of a random sample of the generated labels, (ii) an agreement study against human annotations for a subset, and/or (iii) evaluation on existing human-annotated datasets (e.g., COCO, Visual Genome, OpenImages) for object and relation tasks. Without such an anchor, the 'grounded' dataset claim is unsupported.","section":"§3.1, §3.2, §7, Tables 1–6"},{"comment":"The abstract explicitly promises 'out-of-domain generalisation' and the introduction highlights it, but the main text and supplementary contain no out-of-domain experiment. All downstream evaluations in §5 are on the Action-Genome test set, which is generated by the same pipeline. Table 1B uses released CLIP/SLIP checkpoints, but these are not evaluated on external data. To support the out-of-domain claim, the authors must add experiments that transfer ScenarioCLIP to independent datasets or tasks (e.g., zero-shot or fine-tuned evaluation on COCO/Visual Genome, or on an existing action recognition benchmark). This is a load-bearing missing piece.","section":"Abstract and §5"},{"comment":"The relation-focused region is a purely spatial construction: an RBF-weighted blend of two SAM masks, with non-focused pixels blurred. There is no evidence that these regions correspond to the semantics of the relation triplet. The relation localization evaluation in §8.1.5 and Table 6 uses the same auto-generated relation masks as supervision, so it cannot validate grounding. The paper should provide a human or external assessment of whether the focused regions meaningfully localize the interaction, or demonstrate performance on a dataset with human-annotated relation regions.","section":"§7, Eq. (4)–(5)"},{"comment":"The dataset's relation class count is 225,609, which is extremely large relative to 615,805 images, and the paper claims the vocabulary was 'cleaned' and 'consolidated.' Yet the predicate classification and scene graph evaluation in Table 5 restrict to a subset of ~700 predicates, suggesting that the full 225k relation set is not being meaningfully evaluated. This inconsistency raises doubts about the quality and usability of the relation taxonomy. The authors should clarify how many relations are actually usable after filtering and whether the huge class count reflects duplicate or near-duplicate relations that survived the curation.","section":"§3.2 and §8.1.4"}],"minor_comments":[{"comment":"The arXiv metadata titles the paper 'SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarking,' but the manuscript's internal title is 'ScenarioCLIP: Pretrained Transferable Visual Language Models and Action-Genome Dataset for Natural Scene Analysis.' The dataset is referred to as 'Action-Genome' throughout the main text. This inconsistency should be resolved in revision.","section":"Title/Abstract"},{"comment":"The 'zero-shot' naming is confusing: Table 1A is described as in-domain pretraining on Action-Genome, and the comparison is against PyramidCLIP that is also fine-tuned on the test distribution. This is not the conventional zero-shot setting used for CLIP-style models. Clarify what 'zero-shot' means here (no fine-tuning on the specific task?) and avoid conflating it with the standard cross-dataset zero-shot evaluation.","section":"§5.2, Table 1"},{"comment":"The ablation of λ_KD is only reported for zero-shot retrieval; the effect on detection and linear probe is not shown for the other λ_KD schedules. The claim that 'KD refines the structure' would be stronger with ablations across tasks.","section":"§5.5 and Table 7"},{"comment":"Several references are incomplete or incorrectly formatted (e.g., [56] 'Viualizing data using t-sne'; [40] Ovis appears in the authors list as 'Lu, Shiyin, et al.' with an incomplete author list). Please check all entries.","section":"References"},{"comment":"The figures are dense and the annotations are small. In particular, Figure 4's embedding-dimension labels (e.g., 'nO×3×512×512' for object crops) are confusing and should be clarified in the caption.","section":"Figure 3/4"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the self-referential evaluation. As a dataset paper, the value proposition depends on annotation quality, but the paper provides no independent verification. I would not recommend rejection because the issues are addressable: add a human-validated subset, evaluate on external human-annotated benchmarks, and add the promised OOD experiments. The title inconsistency and the mismatch between the arXiv title and the internal text suggest the manuscript may be an older draft; the editor may want to ask the authors to align the version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real attempt at a large resource and a model, but the evaluation loop is closed. The same VLM pipeline that produced the labels is the one the model is tested on. Without a human-audited subset or an external benchmark, the \"grounded\" claims do not hold up.\n\nWhat is genuinely new: the 615,805-image dataset with action, object, and relation annotations generated at scale; the pipeline that combines Ovis-Gemma, GroundingDINO, SAM, and RBF-based focused regions; and the tri-level disentangled ScenarioCLIP with EMA distillation. The internal comparison against PyramidCLIP under the same training setup is reasonably controlled, and the ablations of the distillation weight are a plus. The paper is detailed enough to reimplement, which is more than many dataset papers offer.\n\nThe soft spots are load-bearing. Section 3 and the supplementary describe a fully automatic annotation pipeline with no human verification. Every number in Tables 1-6 is measured against those auto-generated labels. If Ovis hallucinates a relation, ScenarioCLIP is rewarded for retrieving that hallucinated label. The one-sentence mention of reusing Kinetics/MMT/OpenPVSG labels is not backed by any numbers, splits, or quality analysis. The abstract promises out-of-domain generalization, but no OOD experiments appear anywhere in the main text or supplementary. The dataset is not actually released, only a GitHub placeholder. And the \"Action-Genome\" name collides with an existing scene-graph dataset that is not cited; the SCLARO title in the submission metadata suggests the authors may already be aware of the problem.\n\nNone of these flaws make the internal model comparison meaningless—ScenarioCLIP may genuinely beat PyramidCLIP on this label distribution. But they remove the basis for the resource claim and make the reported gains untrustworthy as evidence of real scene understanding. The fixes are straightforward: a human-verified subset, agreement statistics against the source datasets, a real OOD evaluation, and a released dataset.\n\nWho is this for? People working on scene-graph and vision-language pretraining who want to see a large auto-labeled dataset and a relation-aware architecture. It deserves a serious referee; the verdict should be conditional, not a desk reject. I would engage with it, but I would not cite it as ground truth until the external validation appears.","headline":"A plausible 615K-image scene dataset and relation-aware CLIP variant, but every evaluation is measured against the same auto-generated labels used to build the dataset, so the 'grounded' claim is not yet supported.","tokens_in":19771,"tokens_out":2464,"would_cite":false,"duration_ms":26001,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper sets out to show that explicit, grounded modeling of actions, objects, and relations—not just global image-text alignment—improves vision-language scene understanding, and backs the claim with a 615,805-image dataset and a tri-lev","keywords":["vision-language pretraining","scene understanding","relation triplets","grounding","object detection","knowledge distillation","zero-shot retrieval","scene graph"],"falsifier":"Take a random sample of a few hundred dataset images, have annotators list the objects actually present and the relations that hold between the detected boxes, and compute agreement with the auto-generated triplets. A large fraction of unsupported or absent-object triplets would show that the grounded claim fails; a companion check would be to train ScenarioCLIP only on human-verified samples and see whether the object and relation retrieval gains survive on unseen object-predicate combinations.","tokens_in":18817,"feed_emoji":"🖼️","tokens_out":10386,"duration_ms":103060,"temperature":0.7,"pith_summary":"This paper argues that dual-stream vision-language models, which align whole images with short captions, miss the compositional structure of real scenes: multiple objects, actions, and inter-object relations cannot be captured by one global embedding. To fix this, it introduces SCLARO, a 615,805-image dataset spanning indoor, outdoor, and driving scenes, where every image carries a global action caption, object bounding boxes, and relation triplets of the form (object1, relation, object2), plus relation-focused regions that localize each interaction. It then proposes ScenarioCLIP, a model with separate visual and text encoders for the global scene, objects, and relations, trained with contrastive alignment at each level and an EMA-based knowledge-distillation objective that transfers global context into local representations. On zero-shot retrieval, linear-probe classification, object detection, predicate classification, and scene-graph classification, the model outperforms a hierarchical shared-encoder baseline, with the largest gains at the object and relation levels (e.g., +14.19 points Top-1 on object zero-shot retrieval). If correct, the work supplies both a large-scale training resource and a concrete recipe for making relational structure explicit in foundation-model pretraining.","feed_headline":"Object-relation modeling lifts zero-shot retrieval by 14 points","feed_subtitle":"Grounded object and relation labels push separated encoders past a shared baseline.","key_machinery":"The load-bearing mechanism is a tri-level disentangled encoder set plus an EMA knowledge-distillation loop. Six encoders—global, object, and relation for vision and text, all initialized from a pretrained vision-language backbone—map the full image, object crops, and relation-focused regions, and their text counterparts, into a shared space. Contrastive alignment is applied separately at the global, object, and relation levels, with synthetic hard negatives formed by swapping objects or replacing predicates with opposites. Simultaneously, an exponential-moving-average teacher transfers knowledge from the global visual embedding to object and relation visual embeddings via KL divergence, whil","core_discovery":"The core claim is that relational scene understanding can be learned by vision-language pretraining if the model is given grounded relational inputs and an architecture that keeps global, object, and relation semantics separate. ScenarioCLIP encodes, per image, the full scene, cropped objects, and relation-focused regions through three visual encoders, and the corresponding action caption, object names, and relation triplets through three text encoders; contrastive losses align each level across modalities, and exponential-moving-average teachers distill global visual semantics into object and relation embeddings (and fine-grained text semantics into the global text embedding). On the Action","pith_inferences":["If the auto-generated annotations are accurate enough, an obvious testable extension is to ablate the hard-negative triplets versus the disentangled encoders separately; knowing which ingredient drives the +14.19 object-retrieval gain would help practitioners decide what to adopt.","Because the paper does not report human verification of the vision-language-generated labels, a before-using-the-dataset sanity check is to re-annotate a random subset with independent annotators and measure agreement; that number would bound how much of the measured gain is real semantics versus label artifacts.","The EMA distillation direction—global-to-local on vision, local-to-global on text—is a recipe that could transfer to other modalities (video, LiDAR, robotics state), where a single caption also under-specifies the scene; scenario-specific grounding would be the test."],"forward_implications":["Explicit object- and relation-level encoders, rather than a single shared encoder, yield large object zero-shot retrieval gains (+14.19 Top-1) without hurting action-level performance, so relational structure can be added to existing vision-language pipelines.","Frozen ScenarioCLIP representations separate action, object, and relation classes better under linear probing (+3.83 to +4.56 Top-1), implying cleaner class boundaries at every level.","The relation encoder transfers to object detection, where using the global encoder for proposals and the object encoder for RoI features nudges AP from 9.7 to 9.9 (AP50 18.6 to 18.9), and to predicate/scene-graph classification (R@1 gains of 4.51 and 5.70).","The dataset itself, with 225,609 relation classes and focused regions for each triplet, provides a large-scale benchmark for future work on grounded scene understanding, and the synthetic hard negatives give a ready-made contrastive training signal."],"fun_headline_variants":["Disentangled encoders lift scene-relation understanding","Grounded relation triplets improve zero-shot scene retrieval","SCLARO: 615k images with scene, object, relation labels","Separate encoders for scene, object, relation beat shared ones"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire benchmark rests on auto-generated annotations being true: the action captions, object boxes, and relation triplets come from a vision-language model plus detection and segmentation tools, and the paper reports no human verification or independent agreement check, so every measured improvement is computed against labels that could contain hallucinated objects, wrong relations, or mislocalized boxes.","fun_headline_variants_meta":{"raw":{"variants":["Disentangled encoders lift scene-relation understanding","Grounded relation triplets improve zero-shot scene retrieval","SCLARO: 615k images with scene, object, relation labels","Separate encoders for scene, object, relation beat shared ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000391,"raw_usage":{"total_tokens":1908,"prompt_tokens":772,"completion_tokens":1136,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1064}},"tokens_in":516,"tokens_out":1136,"duration_ms":12391,"temperature":1.0,"reasoning_tokens":1064,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:17:58.067753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of a few hundred dataset images, have annotators list the objects actually present and the relations that hold between the detected boxes, and compute agreement with the auto-generated triplets. A large fraction of unsupported or absent-object triplets would show that the grounded claim fails; a companion check would be to train ScenarioCLIP only on human-verified samples and see whether the object and relation retrieval gains survive on unseen object-predicate combinations.","supporting_citations":[],"review_version":1}