{"id":"93ca270a-0497-4fd3-ac12-8c72e15c0c55","arxiv_id":"2411.15933","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Localizing the foreground before classification and fusing its classifier output with the full-image prediction improves accuracy and robustness to background shifts in supervised and zero-shot VLM recognition.","lead":"The paper proposes L2R2, a recognition pipeline that first localizes the object with a zero-shot detector, classifies the cropped object and the full image separately, and then fuses the two predictions. It reports accuracy and robustness gains on fine-grained datasets and vision-language models, though the largest ImageNet gains rely on ground-truth prompts for localization.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim overreaches: zero-shot localization is only automatic on fine-grained datasets; ImageNet gains rely on oracle GT prompts, and fully automatic HIN FG is worse than FULL.","rationale":"The reader's weakest_assumption identifies exactly the oracle-prompt issue and the failure of fully automatic HIN to confirm the FG advantage. This is the most load-bearing concern because the paper's novelty is explicitly 'zero-shot' localization, yet the general-object experiments rely on ground-truth class labels to generate masks. The abstract and contributions claim localization before recognition is possible 'for a wide range of datasets,' but the evidence supports that claim only for fine-grained datasets with a single dominant object, where meta-prompts like 'dog' or 'mushroom' suffice. On ImageNet and HIN, the oracle leaks the answer to the localization module, so the recognition gain cannot be attributed to zero-shot decomposition. The fully automatic HIN experiment (Table 7 'No GT') shows FG alone is 1.78 points below FULL on the original split and roughly tied on the shifted splits, meaning the method's standalone FG baseline fails precisely in the general setting where the paper concedes (Limitations) that current zero-shot detectors are insufficient. This does not invalidate the fine-grained results, including the VLM doubling on FungiTastic, so a conditional verdict remains appropriate: the authors must either restrict their claims to fine-grained single-object benchmarks or provide a fully automatic general-object experiment where FG and L2R2 fusion beat FULL. The requested check would settle whether the general-object claim can survive without oracle prompts.","tokens_in":28177,"tokens_out":4827,"duration_ms":45334,"concrete_test":"Re-run the ImageNet-1K experiments from Table 1 replacing oracle prompts with the fully automatic top-k prompting protocol of Table 7 'No GT', and report FG and FG+FULL fusion accuracy on the original validation set. If FG does not exceed FULL and the fusion gain over FULL is not statistically significant, then the oracle prompts, not zero-shot localization, are responsible for the reported gains on general-object datasets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines the decomposition as xFG, xBG = fD(x, p), with p a dataset-specific text prompt, and the paper emphasizes that detection is zero-shot. The central claim that this enables localization-before-recognition 'for a wide range of datasets' is undercut by two facts. First, for ImageNet-1K and Hard ImageNet, Section 4.1 states 'Oracle prompts: ... we pre-compute masks ... by prompting each image with the text of its GT label.' This leaks the class identity into the localization stage; the downstream classifier then 'recognizes' an object whose class was already supplied to the detector. Second, in the only fully automatic general-object experiment (Hard ImageNet, Table 7 'No GT', where the FULL model's top-k predictions generate prompts), the FG-only model achieves 95.55% versus 97.33% for FULL on the original split, and 81.24% versus 81.33% on LT, 90.10% versus 90.51% on CT. Thus, without oracle prompts, foreground localization does not provide a strong baseline in a multi-object setting. The reported 7.11% average FG gain (Table 1) is dominated by Spawrious, where the BG shift is extreme and synthetic, and by oracle-prompted HIN and ImageNet columns. The method's demonstrated automatic benefit is therefore restricted to fine-grained datasets with a single dominant object, not to general object recognition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes L2R2, a recognition pipeline that first localizes the foreground (FG) with a zero-shot text-prompted detector (GroundingDINO or OWLv2), then classifies the FG crop and the full image (FULL) with either independently trained supervised classifiers or a shared VLM, and finally fuses the two predictions with confidence-based, threshold-based, or learned weighted-logit fusion. The authors evaluate on Hard ImageNet (including two new long-tail and constant-background test sets), Stanford Dogs, Spawrious, ImageNet-1K, FungiTastic, and CounterAnimal. They report that the FG-only model improves over FULL by 7.11% on average in supervised settings and that FG⊕FULL fusion improves several VLM baselines, doubling BioCLIP's FungiTastic accuracy from about 19% to 38%. For ImageNet and Hard ImageNet, masks are produced by prompting the detector with ground-truth labels (\"oracle prompts\"); fully automatic masks are evaluated on Hard ImageNet in the appendix. The paper is an empirical study and contains no theoretical derivation.","tokens_in":127,"tokens_out":8099,"duration_ms":125673,"significance":"If the results are scoped correctly, the paper makes a useful contribution: it demonstrates that foreground-only classification is a strong baseline for background-shift robustness (notably on Spawrious, where it outperforms published DG methods), that fusing FG with FULL can recover in-domain accuracy without sacrificing robustness, and that the recipe transfers to zero-shot VLM classification with a parameter-free confidence fusion. The authors are transparent about the oracle-prompting protocol on ImageNet/HIN and state in the Limitations that current zero-shot detectors do not support a fully general multi-object setting. However, several headline claims overreach the evidence: the 7.11% average gain is dominated by one synthetic dataset and by oracle-prompted columns, and the automatic HIN experiment in Table 7 shows FG below FULL. The paper would be significantly strengthened by reframing the general-object results as oracle upper bounds and reserving the \"wide range\" claim for fine-grained, single-dominant-object datasets.","major_comments":[{"comment":"The oracle-prompting protocol for ImageNet and Hard ImageNet provides the class identity to the localization stage. Section 3.1 states that masks are pre-computed \"by prompting each image with the text of its GT label,\" and Tables 1 and 5 mark these columns with an asterisk. A recognition pipeline that can prompt the detector with the ground-truth class has effectively been given the answer before classification, so these columns cannot support the abstract's claim that zero-shot foreground localization \"is possible for a wide range of datasets.\" I recommend presenting the ImageNet/HIN columns explicitly as oracle upper-bound results, separate from the automatic fine-grained results, and adjusting the abstract and Section 5.1 accordingly.","section":"Section 3.1 and Tables 1, 5"},{"comment":"The statement that \"the FG model outperforms FULL by 7.11% on average\" is not representative of the fully automatic experiments. In Table 1, only the Stanford Dogs (+0.97), Spawrious (+48.11), and FungiTastic (-0.08) columns use automatic detection; the three HIN columns and the ImageNet columns use oracle prompts. Excluding the oracle columns and the extreme synthetic Spawrious shift, the average FG gain over the two remaining automatic fine-grained columns is about +0.45%. The 7.11% figure should be disaggregated (automatic vs. oracle, with and without Spawrious) so the reader can see where the benefit actually comes from.","section":"Section 5.1, Table 1"},{"comment":"In the only fully automatic general-object experiment, FG localization does not provide a strong baseline. Table 7 (\"No GT\") shows FG at 95.55% vs. FULL at 97.33% on the original HIN split, 81.24% vs. 81.33% on LT, and 90.10% vs. 90.51% on CT; FG is below FULL on all three splits. This is in tension with Section 5.1's claim that FG \"either improves or maintains performance around the FULL baseline on all datasets.\" The Limitations paragraph already concedes that current zero-shot detectors do not allow a fully general ImageNet setup, but the central claims in the abstract and Section 5.1 should be revised to reflect this negative automatic result rather than only the oracle-based columns.","section":"Appendix C.1, Table 7"},{"comment":"The introduction's claim that \"the method consistently improves the performance of diverse CLIP-like models on all datasets\" is not supported by Table 5. For SigLIP2, FG⊕maxFULL decreases accuracy on the CounterAnimal \"rare\" test set by 1.16% (from 89.36 to 88.20), and FG-only is worse than FULL on most SigLIP2 columns. Section 5.2 correctly notes the rare-set exception, so the wording in Section 1 and the abstract should be aligned with the actual pattern: consistent small gains on most, but not all, evaluated test sets.","section":"Section 1 and Section 5.2, Table 5"}],"minor_comments":[{"comment":"There are several typos and small errors: \"suuppression\" in Related Work, \"detc-tions\" in Appendix B.1, \"becuase\" in Section 5.2, \"FungiTatsic\" in the Table 4 caption, \"Imagent\" in Appendix C.1, \"Resutls\" in the Table 6 caption, and \"43-77\" in Table 3 (which should presumably be \"43.77\").","section":"Throughout"},{"comment":"The new HIN-LT (226 images) and HIN-CT (99 images) test sets are small, but the main-text Table 1 reports point estimates without confidence intervals; differences on the order of 0.5-1% should therefore be interpreted cautiously.","section":"Section 4.1 / Appendix A"},{"comment":"The repeated citation sequence \"[4, 4, 16, 16, 45, 45, 55, 55, 59, 59]\" appears to be a formatting error and should be deduplicated.","section":"Section 1"},{"comment":"In the text, references to Figures 4 and 5 appear as \"presented in 4\" and \"shown in 5\"; the word \"Figure\" should be added.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The authors are honest in the Limitations section, but the abstract and contribution list oversell the automatic scope of the method. I believe the paper can be fixed with reframing: oracle columns should be presented as upper bounds, the 7.11% average should be disaggregated, and the \"wide range\" claim should be restricted to fine-grained datasets. There are no concerns about citation behavior or novelty disclosure from my side."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: L2R2 is a competent, mostly well-run empirical paper. It shows that cropping to the detected foreground and fusing that prediction with the full-image prediction helps on fine-grained benchmarks and with CLIP/SigLIP-style models. The new bit is real: the earlier FG/BG decoupling work used ground-truth masks or expensive edge proposals, and the paper shows modern zero-shot detectors make the pipeline practical for single-dominant-object domains.\n\nThe strongest results are on Spawrious, where the FG baseline crushes the domain-generalization baselines, and on the VLM evaluation, especially BioCLIP on FungiTastic. The multi-seed supervised experiments, the ensembling control, and the explicit caveats in the limitations section all suggest the authors are being straight with the data.\n\nThe soft spots are about scope, not about whether the method can help. The abstract and Section 5.1 lean on a 'wide range of datasets' claim, but the ImageNet/Hard-ImageNet columns use oracle prompts: the detector is told the ground-truth label per image. That leaks class identity into localization, so those columns do not support the automatic-localization claim. The fully automatic Hard ImageNet experiment (Table 7, No GT) actually shows FG below FULL on the original split (95.55 vs 97.33), so the method's automatic benefit is demonstrated for fine-grained, single-object settings, not general object recognition. The conclusion also quotes 94.39% on Spawrious, which I cannot match to Table 2 (FG is 95.00-95.59 depending on variant). That needs correcting. And the promised code is still just a promise; given the method is simple, code would materially help the reader.\n\nNone of this sinks the core evidence. The fine-grained results and the VLM gains stand on their own. The circularity burden is low because the fusion weights are fit on validation and the robustness claims are tested on held-out shifted sets. The citation pattern is fine, with prior FG/BG work properly credited.\n\nWho it's for: anyone doing fine-grained recognition or robustness for VLMs, and people working on zero-shot detection as a preprocessing step. It deserves a serious referee; I would send it out. The revision needs a clear separation of oracle-prompt from automatic results in the abstract and main text, a corrected Spawrious number, and released code.","headline":"Solid empirical demonstration that zero-shot localization helps fine-grained recognition and VLMs, but the 'wide range of datasets' claim overreaches because the general ImageNet results rely on oracle GT-prompted detection.","tokens_in":29034,"tokens_out":1958,"would_cite":true,"duration_ms":18527,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot foreground localization before recognition, fused with context-aware full-image prediction, makes classifiers both more accurate and more robust to background shifts.","keywords":["object recognition","foreground-background decomposition","zero-shot object detection","background robustness","context-aware classification","vision-language models","domain generalization","fine-grained recognition"],"falsifier":"Run the full L2R2 pipeline on ImageNet with fully automatic localization, with the detection prompt derived from the full-image model's own top-k predictions, and check whether the fused accuracy still exceeds the full-image baseline: Table 7 already reports such a no-ground-truth setting on Hard ImageNet where this is not achieved, with FG at 95.55% versus FULL at 97.33%.","tokens_in":27895,"feed_emoji":"🔍","tokens_out":3183,"duration_ms":27649,"temperature":0.7,"pith_summary":"The paper proposes the L2R2 (Localize to Recognize Robustly) pipeline, which uses a zero-shot text-prompted detector to crop or mask the foreground before classification, then fuses the foreground-only prediction with the standard context-aware full-image prediction. The central claim is that localization can be an integral part of recognition rather than a post-classification step, and that keeping the context model alongside the foreground model gives the best of both worlds: the robustness of background suppression plus the accuracy of context. A sympathetic reader should care because the paper reframes background as useful information to be exploited safely, not merely a nuisance to be suppressed, and shows the same recipe works for both supervised classifiers and large vision-language models.","feed_headline":"Locate the object first, then classify: accuracy and robustness both rise","feed_subtitle":"Zero-shot foreground detection plus context fusion beats full-image models and doubles a fungus-classification VLM's accuracy.","key_machinery":"The central object is the image decomposition operation $x_{FG}, x_{BG} = f_D(x, p)$, where a zero-shot open-vocabulary detector such as GroundingDINO or OWLv2 is prompted with a text phrase $p$ to localize the foreground, producing foreground and background inputs before any classification happens. This decomposition enables a recognition pipeline in which the foreground model $\\Phi(x_{FG})$ and the context-aware full-image model $\\Omega(x_{FULL})$ are trained or evaluated independently, and their outputs are combined by simple fusion operators: maximum confidence $\\oplus_{max}$, robust thresholding $\\oplus_R$, or weighted-logit combination $\\oplus_{WL}$.","core_discovery":"The paper establishes that localizing the foreground before recognition, then fusing the foreground prediction with the full-image context-aware prediction, improves accuracy and robustness across a wide range of benchmarks. The foreground-only model alone outperforms full-image classification by 7.11% on average in the supervised setup, and the fusion further improves in-domain performance without losing robustness to background distribution shifts. For vision-language models, the same parameter-free fusion consistently improves zero-shot classification, most dramatically doubling BioCLIP accuracy on the FungiTastic benchmark from roughly 19% to 38%. The work also introduces two new Hard ImageNet test sets with long-tail and constant backgrounds where the method's robustness is demonstrated.","pith_inferences":["If localization-before-recognition is adopted, progress in open-vocabulary detection will translate directly into recognition gains, making the method's benefits on general multi-object datasets grow as detectors improve.","The large gap between ground-truth-prompt results and the automatic top-k-prompt results on Hard ImageNet suggests that recognition accuracy on such datasets is gated primarily by localization quality rather than by classification capacity.","Because the foreground model is by construction free of background shortcuts, the decomposition could serve as a diagnostic tool for isolating which classification errors are caused by background reliance.","The independence of the two models means specialized background modellers - such as occlusion-aware or generation-based context models - can be plugged into the fusion step without retraining the foreground classifier."],"forward_implications":["Foreground-cropped classification is a strong robustness baseline: it outperforms or matches full-image classification on all supervised benchmarks, and on Spawrious beats all twelve compared domain-generalization methods without any special training procedure.","Fusing foreground and full-image predictions yields in-domain accuracy gains over the foreground model while preserving robustness to background shifts, provided a robust fusion rule such as thresholding is used.","Zero-shot detection can serve as a practical foreground localizer for fine-grained recognition, where a generic meta-prompt such as 'mushroom' or 'dog' suffices for all classes in the dataset.","The same localization-plus-fusion recipe improves zero-shot vision-language classifiers with a parameter-free fusion rule, with consistent gains across CLIP, CLIP-L, BioCLIP, and SigLIP2 models.","Background information remains available and useful: there exist images where the background model alone is correct while both the foreground and full-image models fail, so suppressing background entirely discards useful signal."],"supporting_citations":[{"why":"GroundingDINO is the zero-shot open-vocabulary detector used for foreground localization in most experiments.","marker":"[26]"},{"why":"OWL and OWLv2 are the zero-shot detectors used on Hard ImageNet where GroundingDINO produces too many false positives.","marker":"[30, 31]"},{"why":"Hard ImageNet supplies the segmentation masks and the strong foreground-background correlation benchmark that is extended with the new Long Tail and Constant test sets.","marker":"[33]"},{"why":"Spawrious is the synthetic domain-generalization benchmark with adversarial background shifts against which L2R2 is compared with twelve prior methods.","marker":"[29]"},{"why":"SAM is used to convert detected bounding boxes into segmentation masks for the background-with-shape inputs.","marker":"[21]"},{"why":"FungiTastic supplies the challenging fine-grained fungi benchmark where L2R2 doubles BioCLIP accuracy and where tabular background metadata is fused.","marker":"[38]"},{"why":"CLIP is the foundational vision-language model class whose zero-shot classification L2R2 is applied to and improves.","marker":"[40]"},{"why":"SigLIP2 is the strongest vision-language model evaluated in the main zero-shot experiments.","marker":"[52]"},{"why":"BioCLIP is the domain-specific vision-language model whose FungiTastic performance doubles from about 19% to 38% under L2R2 fusion.","marker":"[48]"},{"why":"The ImageNet-Hard study of foreground-versus-background classifier disagreement motivates the independent two-model fusion approach.","marker":"[50]"}],"fun_headline_variants":["Localize first, then classify: robust and accurate","Zero-shot detection boosts recognition robustness","Context fusion doubles VLM accuracy on fungi","Foreground-first recognition beats full-image models","L2R2: locate before you recognize for robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the zero-shot detector can localize the actual foreground object from a text prompt without knowing the ground-truth class, and this assumption only holds reliably for fine-grained or single-object datasets, while on general multi-object datasets like ImageNet the paper has to bypass it by prompting with each image's ground-truth label in what it calls oracle prompts.","fun_headline_variants_meta":{"raw":{"variants":["Localize first, then classify: robust and accurate","Zero-shot detection boosts recognition robustness","Context fusion doubles VLM accuracy on fungi","Foreground-first recognition beats full-image models","L2R2: locate before you recognize for robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000105,"raw_usage":{"total_tokens":982,"prompt_tokens":841,"completion_tokens":141,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":72}},"tokens_in":457,"tokens_out":141,"duration_ms":2243,"temperature":1.0,"reasoning_tokens":72,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:43:26.576054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full L2R2 pipeline on ImageNet with fully automatic localization, with the detection prompt derived from the full-image model's own top-k predictions, and check whether the fused accuracy still exceeds the full-image baseline: Table 7 already reports such a no-ground-truth setting on Hard ImageNet where this is not achieved, with FG at 95.55% versus FULL at 97.33%.","supporting_citations":[{"cited_title":"Hard ima- genet: Segmentations for objects with strong spurious cues","cited_arxiv_id":null,"evidence_quote":"Hard ImageNet supplies the segmentation masks and the strong foreground-background correlation benchmark that is extended with the new Long Tail and Constant test sets."},{"cited_title":"Spawrious: A benchmark for fine control of spurious correlation biases, 2023","cited_arxiv_id":null,"evidence_quote":"Spawrious is the synthetic domain-generalization benchmark with adversarial background shifts against which L2R2 is compared with twelve prior methods."},{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"SAM is used to convert detected bounding boxes into segmentation masks for the background-with-shape inputs."},{"cited_title":"Bioclip: A vision foundation model for the tree of life","cited_arxiv_id":null,"evidence_quote":"BioCLIP is the domain-specific vision-language model whose FungiTastic performance doubles from about 19% to 38% under L2R2 fusion."},{"cited_title":"Imagenet-hard: The hard- est images remaining from a study of the power of zoom and spatial biases in image classification","cited_arxiv_id":null,"evidence_quote":"The ImageNet-Hard study of foreground-versus-background classifier disagreement motivates the independent two-model fusion approach."}],"review_version":1}