Pith. sign in

REVIEW 4 major objections 4 minor 77 references

Bringing the Context Back into Object Recognition, Robustly

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Zero-shot foreground localization before recognition, fused with context-aware full-image prediction, makes classifiers both more accurate and more robust to background shifts.

desk verdict Solid empirical demonstration that zero-shot localization helps fine-grained recognition and VLMs, but the 'wide range of datasets' claim overreaches because the general ImageNet results rely on oracle GT-prompted detection. read the letter →

arxiv 2411.15933 v2 pith:7SHHAJ7Z submitted 2024-11-24 cs.CV

Klara Janouskova , Cristian Gavrus , Jiri Matas This is my paper · ORCID
classification cs.CV
keywords objectrecognitionforeground-backgrounddecompositionzero-shotdetectionbackgroundrobustnesscontext-awareclassificationvision-languagemodelsdomaingeneralizationfine-grained
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes the L2R2 (Localize to Recognize Robustly) pipeline, which uses a zero-shot text-prompted detector to crop or mask the foreground before classification, then fuses the foreground-only prediction with the standard context-aware full-image prediction. The central claim is that localization can be an integral part of recognition rather than a post-classification step, and that keeping the context model alongside the foreground model gives the best of both worlds: the robustness of background suppression plus the accuracy of context. A sympathetic reader should care because the paper reframes background as useful information to be exploited safely, not merely a nuisance to be suppressed, and shows the same recipe works for both supervised classifiers and large vision-language models.

What carries the argument

The central object is the image decomposition operation $x_{FG}, x_{BG} = f_D(x, p)$, where a zero-shot open-vocabulary detector such as GroundingDINO or OWLv2 is prompted with a text phrase $p$ to localize the foreground, producing foreground and background inputs before any classification happens. This decomposition enables a recognition pipeline in which the foreground model $\Phi(x_{FG})$ and the context-aware full-image model $\Omega(x_{FULL})$ are trained or evaluated independently, and their outputs are combined by simple fusion operators: maximum confidence $\oplus_{max}$, robust thresholding $\oplus_R$, or weighted-logit combination $\oplus_{WL}$.

What would settle it

Run the full L2R2 pipeline on ImageNet with fully automatic localization, with the detection prompt derived from the full-image model's own top-k predictions, and check whether the fused accuracy still exceeds the full-image baseline: Table 7 already reports such a no-ground-truth setting on Hard ImageNet where this is not achieved, with FG at 95.55% versus FULL at 97.33%.

Watch

Extended reading notes

Core claim

The paper establishes that localizing the foreground before recognition, then fusing the foreground prediction with the full-image context-aware prediction, improves accuracy and robustness across a wide range of benchmarks. The foreground-only model alone outperforms full-image classification by 7.11% on average in the supervised setup, and the fusion further improves in-domain performance without losing robustness to background distribution shifts. For vision-language models, the same parameter-free fusion consistently improves zero-shot classification, most dramatically doubling BioCLIP accuracy on the FungiTastic benchmark from roughly 19% to 38%. The work also introduces two new Hard ImageNet test sets with long-tail and constant backgrounds where the method's robustness is demonstrated.

Load-bearing premise

The method assumes the zero-shot detector can localize the actual foreground object from a text prompt without knowing the ground-truth class, and this assumption only holds reliably for fine-grained or single-object datasets, while on general multi-object datasets like ImageNet the paper has to bypass it by prompting with each image's ground-truth label in what it calls oracle prompts.

Editorial extensions

If this is right

  • Foreground-cropped classification is a strong robustness baseline: it outperforms or matches full-image classification on all supervised benchmarks, and on Spawrious beats all twelve compared domain-generalization methods without any special training procedure.
  • Fusing foreground and full-image predictions yields in-domain accuracy gains over the foreground model while preserving robustness to background shifts, provided a robust fusion rule such as thresholding is used.
  • Zero-shot detection can serve as a practical foreground localizer for fine-grained recognition, where a generic meta-prompt such as 'mushroom' or 'dog' suffices for all classes in the dataset.
  • The same localization-plus-fusion recipe improves zero-shot vision-language classifiers with a parameter-free fusion rule, with consistent gains across CLIP, CLIP-L, BioCLIP, and SigLIP2 models.
  • Background information remains available and useful: there exist images where the background model alone is correct while both the foreground and full-image models fail, so suppressing background entirely discards useful signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If localization-before-recognition is adopted, progress in open-vocabulary detection will translate directly into recognition gains, making the method's benefits on general multi-object datasets grow as detectors improve.
  • The large gap between ground-truth-prompt results and the automatic top-k-prompt results on Hard ImageNet suggests that recognition accuracy on such datasets is gated primarily by localization quality rather than by classification capacity.
  • Because the foreground model is by construction free of background shortcuts, the decomposition could serve as a diagnostic tool for isolating which classification errors are caused by background reliance.
  • The independence of the two models means specialized background modellers - such as occlusion-aware or generation-based context models - can be plugged into the fusion step without retraining the foreground classifier.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes L2R2, a recognition pipeline that first localizes the foreground (FG) with a zero-shot text-prompted detector (GroundingDINO or OWLv2), then classifies the FG crop and the full image (FULL) with either independently trained supervised classifiers or a shared VLM, and finally fuses the two predictions with confidence-based, threshold-based, or learned weighted-logit fusion. The authors evaluate on Hard ImageNet (including two new long-tail and constant-background test sets), Stanford Dogs, Spawrious, ImageNet-1K, FungiTastic, and CounterAnimal. They report that the FG-only model improves over FULL by 7.11% on average in supervised settings and that FG⊕FULL fusion improves several VLM baselines, doubling BioCLIP's FungiTastic accuracy from about 19% to 38%. For ImageNet and Hard ImageNet, masks are produced by prompting the detector with ground-truth labels ("oracle prompts"); fully automatic masks are evaluated on Hard ImageNet in the appendix. The paper is an empirical study and contains no theoretical derivation.

Significance. If the results are scoped correctly, the paper makes a useful contribution: it demonstrates that foreground-only classification is a strong baseline for background-shift robustness (notably on Spawrious, where it outperforms published DG methods), that fusing FG with FULL can recover in-domain accuracy without sacrificing robustness, and that the recipe transfers to zero-shot VLM classification with a parameter-free confidence fusion. The authors are transparent about the oracle-prompting protocol on ImageNet/HIN and state in the Limitations that current zero-shot detectors do not support a fully general multi-object setting. However, several headline claims overreach the evidence: the 7.11% average gain is dominated by one synthetic dataset and by oracle-prompted columns, and the automatic HIN experiment in Table 7 shows FG below FULL. The paper would be significantly strengthened by reframing the general-object results as oracle upper bounds and reserving the "wide range" claim for fine-grained, single-dominant-object datasets.

major comments (4)
  1. [Section 3.1 and Tables 1, 5] The oracle-prompting protocol for ImageNet and Hard ImageNet provides the class identity to the localization stage. Section 3.1 states that masks are pre-computed "by prompting each image with the text of its GT label," and Tables 1 and 5 mark these columns with an asterisk. A recognition pipeline that can prompt the detector with the ground-truth class has effectively been given the answer before classification, so these columns cannot support the abstract's claim that zero-shot foreground localization "is possible for a wide range of datasets." I recommend presenting the ImageNet/HIN columns explicitly as oracle upper-bound results, separate from the automatic fine-grained results, and adjusting the abstract and Section 5.1 accordingly.
  2. [Section 5.1, Table 1] The statement that "the FG model outperforms FULL by 7.11% on average" is not representative of the fully automatic experiments. In Table 1, only the Stanford Dogs (+0.97), Spawrious (+48.11), and FungiTastic (-0.08) columns use automatic detection; the three HIN columns and the ImageNet columns use oracle prompts. Excluding the oracle columns and the extreme synthetic Spawrious shift, the average FG gain over the two remaining automatic fine-grained columns is about +0.45%. The 7.11% figure should be disaggregated (automatic vs. oracle, with and without Spawrious) so the reader can see where the benefit actually comes from.
  3. [Appendix C.1, Table 7] In the only fully automatic general-object experiment, FG localization does not provide a strong baseline. Table 7 ("No GT") shows FG at 95.55% vs. FULL at 97.33% on the original HIN split, 81.24% vs. 81.33% on LT, and 90.10% vs. 90.51% on CT; FG is below FULL on all three splits. This is in tension with Section 5.1's claim that FG "either improves or maintains performance around the FULL baseline on all datasets." The Limitations paragraph already concedes that current zero-shot detectors do not allow a fully general ImageNet setup, but the central claims in the abstract and Section 5.1 should be revised to reflect this negative automatic result rather than only the oracle-based columns.
  4. [Section 1 and Section 5.2, Table 5] The introduction's claim that "the method consistently improves the performance of diverse CLIP-like models on all datasets" is not supported by Table 5. For SigLIP2, FG⊕maxFULL decreases accuracy on the CounterAnimal "rare" test set by 1.16% (from 89.36 to 88.20), and FG-only is worse than FULL on most SigLIP2 columns. Section 5.2 correctly notes the rare-set exception, so the wording in Section 1 and the abstract should be aligned with the actual pattern: consistent small gains on most, but not all, evaluated test sets.
minor comments (4)
  1. [Throughout] There are several typos and small errors: "suuppression" in Related Work, "detc-tions" in Appendix B.1, "becuase" in Section 5.2, "FungiTatsic" in the Table 4 caption, "Imagent" in Appendix C.1, "Resutls" in the Table 6 caption, and "43-77" in Table 3 (which should presumably be "43.77").
  2. [Section 4.1 / Appendix A] The new HIN-LT (226 images) and HIN-CT (99 images) test sets are small, but the main-text Table 1 reports point estimates without confidence intervals; differences on the order of 0.5-1% should therefore be interpreted cautiously.
  3. [Section 1] The repeated citation sequence "[4, 4, 16, 16, 45, 45, 55, 55, 59, 59]" appears to be a formatting error and should be deduplicated.
  4. [Section 5.1] In the text, references to Figures 4 and 5 appear as "presented in 4" and "shown in 5"; the word "Figure" should be added.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: L2R2's claims are empirical and tested on separate held-out shifts; the oracle-prompted ImageNet/HIN columns are a disclosed scope limitation, not a construction-equivalent prediction.

full rationale

The paper's method is a pipeline (decomposition by zero-shot detection, separate FG/FULL classifiers, fusion), not a derivation whose conclusions are encoded in its inputs. The main FG-before-recognition evidence does not require ground-truth class knowledge: fine-grained datasets use class-agnostic meta-prompts such as 'dog', 'mushroom', or averaged 'animal/bird/insect/reptile' embeddings, and the Spawrious, Stanford Dogs, FungiTastic, and CounterAnimal results are obtained automatically. The ImageNet and Hard ImageNet columns marked 'oracle detection' do use GT labels to prompt the detector ('Oracle prompts: ... we pre-compute masks for all the datasets based on prompting each image with the text of its GT label'), and the paper's abstract claim of 'a wide range of datasets' is therefore broader than its fully automatic evidence. This is an overclaim/scope caveat, not a circular reduction: the downstream classifier still must choose among all classes from the crop, and the paper explicitly labels these results as oracle and lists the limitation that zero-shot detectors 'do not allow us to apply the methodology to a fully general setup of datasets like ImageNet.' Fusion temperatures, thresholds, and weights are optimized on validation sets, but the robustness claims are evaluated on separated shifted test sets (HIN-LT, HIN-CT, Spawrious test, CounterAnimal rare), so no fitted parameter is renamed as a prediction. Self-citations ([22], [37], [38], [39]) provide benchmark data or related work; none is load-bearing as a uniqueness theorem or unverified premise, and the cited datasets are externally published benchmarks. No circular step was found.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No invented entities are introduced. The free parameters are standard calibration/fusion hyperparameters fitted on train or validation data; they do not themselves encode the robustness result, and the Spawrious failure mode shows the fitting does not force the reported gains.

free parameters (5)
  • Temperature scaling T per base model = optimized on validation set (not reported)
    Used in eq. (2) to calibrate confidences for confidence-based fusion; the value is dataset- and model-specific and affects which model wins in max-scaled-confidence fusion.
  • Threshold t for ⊕R fusion = optimized on validation set
    Controls when FG prediction is trusted over FULL/BG; on Spawrious it is critical because validation does not contain shifted backgrounds.
  • Weighted logits w1, w2 (per class) = trained with cross-entropy on training set
    Class-dependent combination weights for fusion model 9; on Spawrious these weights overfit to training BG correlations (test 27.71% vs 91.25% for robust fusion).
  • Top-k for automatic HIN masking = optimized on validation set (k not explicitly stated)
    Number of FULL model predictions used to prompt the detector in the fully automatic setup; influences precision/recall of FG masks.
  • Weighted average alpha in TempScaled WeightedAvg = optimized on validation set
    Fusion weight for averaging scaled logits in B.3, model 6.
assumptions (4)
  • domain assumption Zero-shot detectors (GroundingDINO, OWLv2) can localize the target foreground from a dataset-specific text prompt.
    Invoked in Section 3.1, equation (1). The paper's own HIN automatic experiment shows this fails on general multi-object images, which is why ImageNet uses oracle prompts.
  • domain assumption A generic meta-class prompt (e.g., 'dog', 'mushroom') is sufficient to localize objects in fine-grained datasets.
    Section 3.1 and Appendix B.1 state this for Stanford Dogs, Spawrious, and FungiTastic; it breaks if the meta-class is not visually coherent (e.g., 'animal' for CounterAnimal requires averaging multiple prompts).
  • domain assumption Confidence scores of base classifiers are comparable after temperature scaling, enabling max-confidence fusion.
    Section 3.3 and B.3 rely on this; the poor Spawrious performance of ⊕max fusion (25.9% for FG⊕BG, Table 14) shows the assumption fails when training and test backgrounds differ.
  • domain assumption Object crops preserve the class-discriminative information; padding to a square with a constant value does not distort recognition.
    Section 4.2 describes FG inputs as cropped bounding boxes padded to a square; if the detector box is imprecise (thin spiders in CounterAnimal), FG inputs lose information.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bringing the Context Back into Object Recognition, Robustly." pith.science (2026). https://pith.science/paper/7SHHAJ7Z

@misc{pith2026241115933,
  author       = {Pith},
  title        = {Pith review of: Bringing the Context Back into Object Recognition, Robustly},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7SHHAJ7Z}},
  note         = {Machine review of arXiv:2411.15933}
}
read the original abstract

In object recognition, both the subject of interest (referred to as foreground, FG, for simplicity) and its surrounding context (background, BG) may play an important role. However, standard supervised learning often leads to unintended over-reliance on the BG, limiting model robustness in real-world deployment settings. The problem is mainly addressed by suppressing the BG, sacrificing context information for improved generalization. We propose "Localize to Recognize Robustly" (L2R2), a novel recognition approach which exploits the benefits of context-aware classification while maintaining robustness to distribution shifts. L2R2 leverages advances in zero-shot detection to localize the FG before recognition. It improves the performance of both standard recognition with supervised training, as well as multimodal zero-shot recognition with VLMs, while being robust to long-tail BGs and distribution shifts. The results confirm localization before recognition is possible for a wide range of datasets and they highlight the limits of object detection on others

Figures

Figures reproduced from arXiv: 2411.15933 by the authors.

Figure 1
Figure 1. The complementarity of foreground (FG) and background (BG) in recognition. The standard approach, background suppression, makes correct identification in (a) nearly impossible, and difficult in (b); the spectacled bear is the most herbivorous of all bear species. On the other hand, rare backgrounds with possibly huge diversity hurt classification – (d) shows a cheetah after a snowfall in South Africa, not a snow leo… view at source ↗
Figure 2
Figure 2. VLM (CLIP-B) – zero-shot recognition with ground [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The “Localize to Recognize Robustly” approach to [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The unexpected role of shape in BG modelling. When in￾vestigating the results on the Hard ImageNet dataset, many exam￾ples where found where full image prediction is incorrect but both FG and BG (with shape) predictions are correct Possible explana￾tion: the mask provi…
Figure 5
Figure 5. Figure 5: Examples where FG model is correct and both full image and BG models are incorrect on Hard ImageNet - Long Tail. ERM [53] +6.14 77.49 JTT [25] +18.89 90.24 GroupDRO [44] +9.23 80.58 Mixup [60] +17.13 88.48 IRM [C] +4.10 75.45 Mixup [62] +17.29 88.64 CORAL [49] +18.31 8…
Figure 6
Figure 6. Figure 6: Images from the two new test sets for Hard ImageNet - [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Common problems in the counter animal dataset. Each row shows a random sample of images from a class/ [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: The relative role of FG and BG for the 215 FungiTastic classes shown by the weights of the learned weighted logits combination model, i.e. Model 9. in Section B.3. The BGs has a higher weight for about 15% FULL FGC FGM BGS BGB [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Different kinds of input to the FG and BG models. FULL is the standard full image, FGC is cropped based on the segmentation bounding box, FGM is same as FGC but with the BG regions masked out, BGS is the shape-preserving BG model with FG regions masked out and BGB has …
Figure 10
Figure 10. Figure 10: Extreme [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 12
Figure 12. Figure 12: Per-class accuracy % increase or decrease w.r.t. full [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 55 canonical work pages

  1. [1]

    Detecting out-of-context objects using contextual cues

    Manoj Acharya, Anirban Roy, Kaushik Koneripalli, Susmit Jha, Christopher Kanan, and Ajay Divakaran. Detecting out-of-context objects using contextual cues. arXiv preprint arXiv:2202.05930, 2022. 1, 3

    work page Pith review arXiv 2022

  2. [2]

    Masking strategies for background bias removal in computer vision models

    Ananthu Aniraj, Cassio F Dantas, Dino Ienco, and Diego Marcos. Masking strategies for background bias removal in computer vision models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4397– 4405, 2023. 1, 3

    work page 2023

  3. [3]

    Masktune: Mitigating spurious correlations by forcing to explore

    Saeid Asgari, Aliasghar Khani, Fereshte Khani, Ali Gho- lami, Linh Tran, Ali Mahdavi Amiri, and Ghassan Hamarneh. Masktune: Mitigating spurious correlations by forcing to explore. Advances in Neural Information Process- ing Systems, 35:23284–23296, 2022. 3

    work page 2022

  4. [4]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Ad- vances in neural information processing systems , 32, 2019. 1, 2, 3

    work page 2019

  5. [5]

    Bird- snap: Large-scale fine-grained visual categorization of birds

    Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. Bird- snap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2011–2018, 2014. 4, 7

    work page 2011

  6. [6]

    Are we done with ImageNet? 2020

    Lucas Beyer, Olivier J H ´enaff, Alexander Kolesnikov, Xiao- hua Zhai, A ¨aron Van Den Oord, Google Brain, and Deep- mind ( London. Are we done with ImageNet? 2020. 13

    work page 2020

  7. [7]

    Mitigating the effect of incidental cor- relations on part-based learning

    Gaurav Bhatt, Deepayan Das, Leonid Sigal, and Vineeth N Balasubramanian. Mitigating the effect of incidental cor- relations on part-based learning. Advances in Neural Infor- mation Processing Systems, 36, 2024. 1, 3

    work page 2024

  8. [8]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022. 3

    2022

Show all 77 references
  1. [9]

    Putting the object back into video object segmentation

    Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3151–3161, 2024. 3

    2024

  2. [10]

    Invariant causal mechanisms through distribu- tion matching

    Mathieu Chevalley, Charlotte Bunne, Andreas Krause, and Stefan Bauer. Invariant causal mechanisms through distribu- tion matching. arXiv preprint arXiv:2206.11646, 2022. 6

    2022 arXiv

  3. [11]

    Fine-grained visual classification with high-temperature re- finement and background suppression

    Po-Yung Chou, Yu-Yung Kao, and Cheng-Hung Lin. Fine-grained visual classification with high-temperature re- finement and background suppression. arXiv preprint arXiv:2303.06442, 2023. 1, 3

    2023 arXiv

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 13

    2009

  5. [13]

    An empirical study of context in object detection

    Santosh K Divvala, Derek Hoiem, James H Hays, Alexei A Efros, and Martial Hebert. An empirical study of context in object detection. In 2009 IEEE Conference on computer 9 vision and Pattern Recognition , pages 1271–1278. IEEE,

    2009

  6. [14]

    Network calibration by class-based temperature scaling

    Lior Frenkel and Jacob Goldberger. Network calibration by class-based temperature scaling. In 2021 29th European Sig- nal Processing Conference (EUSIPCO) , pages 1486–1490. IEEE, 2021. 4, 14

    2021

  7. [15]

    Shortcut learning in deep neural networks

    Robert Geirhos, J ¨orn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Fe- lix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. 1

    2020

  8. [16]

    Aspire: Language-guided data augmentation for improving robustness against spurious correlations

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, S Sakshi, Sanjoy Chowdhury, and Dinesh Manocha. Aspire: Language-guided data augmentation for improving robustness against spurious correlations. In Find- ings of the Association for Computational Linguistics...

    2024

  9. [17]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR,

  10. [18]

    The two dimensions of worst-case training and their integrated effect for out-of-domain generalization

    Zeyi Huang, Haohan Wang, Dong Huang, Yong Jae Lee, and Eric P Xing. The two dimensions of worst-case training and their integrated effect for out-of-domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 9631–9641, 2022. 6

    2022

  11. [19]

    Segment anything in high qual- ity

    Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi- Keung Tang, Fisher Yu, et al. Segment anything in high qual- ity. Advances in Neural Information Processing Systems, 36,

  12. [20]

    Novel dataset for fine-grained image categorization

    Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, 2011. 5, 13

    2011

  13. [21]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 202...

    2023

  14. [22]

    Flaws of imagenet, computer vision’s favourite dataset

    Nikita Kisel, Illia V olkov, Katerina Hanzelkova, Klara Janouskova, and Jiri Matas. Flaws of imagenet, computer vision’s favourite dataset. arXiv preprint arXiv:2412.00076,

    arXiv

  15. [23]

    Out-of-distribution general- ization via risk extrapolation (rex)

    David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution general- ization via risk extrapolation (rex). In International confer- ence on machine learning, pages 5815–5826. PMLR, 2021. 6

    2021

  16. [24]

    Domain generalization with adversarial feature learning

    Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400–5409, 2018. 6

    2018

  17. [25]

    Just train twice: Improving group robustness without training group information

    Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghu- nathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Confer- ence on Machine Learning, pages 6781–6792. PMLR, 2021. 6

    2021

  18. [26]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 2, 3, 4, 14

    2023 arXiv

  19. [27]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2022. 5

    2022

  20. [28]

    Camouflaged instance seg- mentation via explicit de-camouflaging

    Naisong Luo, Yuwen Pan, Rui Sun, Tianzhu Zhang, Zhi- wei Xiong, and Feng Wu. Camouflaged instance seg- mentation via explicit de-camouflaging. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17918–17927, 2023. 3

    2023

  21. [29]

    Spawrious: A benchmark for fine control of spurious correlation biases, 2023

    Aengus Lynch, Gb `etondji J-S Dovonon, Jean Kaddour, and Ricardo Silva. Spawrious: A benchmark for fine control of spurious correlation biases, 2023. 2, 5, 6, 7, 8, 13, 18

    2023

  22. [30]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pages 728–755. S...

    2022

  23. [31]

    Scaling open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Advances in Neu- ral Information Processing Systems, 36, 2024. 4, 14

    2024

  24. [32]

    A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes

    Mazda Moayeri, Phillip Pope, Yogesh Balaji, and So- heil Feizi. A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1908...

    2022

  25. [33]

    Hard ima- genet: Segmentations for objects with strong spurious cues

    Mazda Moayeri, Sahil Singla, and Soheil Feizi. Hard ima- genet: Segmentations for objects with strong spurious cues. Advances in Neural Information Processing Systems , 35: 10068–10077, 2022. 3, 5, 13, 19

    2022

  26. [34]

    Obtaining well calibrated probabilities using bayesian binning

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, 2015. 14

    2015

  27. [35]

    The role of context in object recognition

    Aude Oliva and Antonio Torralba. The role of context in object recognition. Trends in cognitive sciences, 11(12):520– 527, 2007. 1, 2

    2007

  28. [36]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 9

    2023 arXiv

  29. [37]

    Jeppe- sen, Jacob Heilmann-Clausen, Thomas Læssøe, and Tobias Frøslev

    Luk ´aˇs Picek, Milan ˇSulc, Ji ˇr´ı Matas, Thomas S. Jeppe- sen, Jacob Heilmann-Clausen, Thomas Læssøe, and Tobias Frøslev. Danish fungi 2020 - not just another image recog- nition dataset. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision ...

    2020

  30. [38]

    Fungitastic: A multi-modal dataset and benchmark for image categorization

    Lukas Picek, Klara Janouskova, Milan Sulc, and Jiri Matas. Fungitastic: A multi-modal dataset and benchmark for image categorization. arXiv preprint arXiv:2408.13632, 2024. 2, 4, 13, 16

    2024 arXiv

  31. [39]

    Animal iden- tification with independent foreground and background mod- eling

    Lukas Picek, Lukas Neumann, and Jiri Matas. Animal iden- tification with independent foreground and background mod- eling. arXiv preprint arXiv:2408.12930, 2024. 1, 3

    2024 arXiv

  32. [40]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  33. [41]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 2, 3

    2024 arXiv

  34. [42]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 13

    2022

  35. [43]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015. 2, 3, 4, 13

    2015

  36. [44]

    Distributionally robust neural networks for group shifts: On the importance of regularization for worst- case generalization

    Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst- case generalization. arXiv preprint arXiv:1911.08731, 2019. 6

    1911 arXiv

  37. [45]

    Not using the car to see the sidewalk–quantifying and controlling the effects of context in classification and segmentation

    Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk–quantifying and controlling the effects of context in classification and segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8218–8226...

    2019

  38. [46]

    Gradi- ent matching for domain generalization

    Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradi- ent matching for domain generalization. arXiv preprint arXiv:2104.09937, 2021. 6

    2021 arXiv

  39. [47]

    Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022

    Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022. 1, 13

    2022

  40. [48]

    Bioclip: A vision foundation model for the tree of life

    Samuel Stevens, Jiaman Wu, Matthew J Thompson, Eliza- beth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger- Wolf, et al. Bioclip: A vision foundation model for the tree of life. In Proceedings of the IEEE/CVF conference...

  41. [49]

    Deep coral: Correlation alignment for deep domain adaptation

    Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision– ECCV 2016 Workshops: Amsterdam, The Netherlands, Oc- tober 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 443–450. Springer, 2016. 6

    2016

  42. [50]

    Imagenet-hard: The hard- est images remaining from a study of the power of zoom and spatial biases in image classification

    Mohammad Reza Taesiri, Giang Nguyen, Sarra Habchi, Cor- Paul Bezemer, and Anh Nguyen. Imagenet-hard: The hard- est images remaining from a study of the power of zoom and spatial biases in image classification. Advances in Neural Information Processing Systems, 36, 2024. 1, 3

    2024

  43. [51]

    Contextual priming for object detection

    Antonio Torralba. Contextual priming for object detection. International journal of computer vision, 53:169–191, 2003. 1, 2

    2003

  44. [52]

    Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, ...

    2025 arXiv

  45. [53]

    Principles of risk minimization for learn- ing theory

    Vladimir Vapnik. Principles of risk minimization for learn- ing theory. Advances in neural information processing sys- tems, 4, 1991. 6

    1991

  46. [54]

    When does dough become a bagel? Analyzing the remaining mistakes on ImageNet

    Vijay Vasudevan, Benjamin Caine, Raphael Gontijo-Lopes, Sara Fridovich-Keil, and Rebecca Roelofs. When does dough become a bagel? Analyzing the remaining mistakes on ImageNet. 13

  47. [55]

    Clad: A contrastive learning based approach for background debiasing

    Ke Wang, Harshitha Machiraju, Oh-Hyeon Choung, Michael Herzog, and Pascal Frossard. Clad: A contrastive learning based approach for background debiasing. arXiv preprint arXiv:2210.02748, 2022. 1, 2, 3

    2022 arXiv

  48. [56]

    A sober look at the robustness of clips to spurious features

    Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. A sober look at the robustness of clips to spurious features. Advances in Neural Information Processing Systems, 37:122484–122523, 2025. 2, 3, 5, 13

    2025

  49. [57]

    Pytorch image models

    Ross Wightman. Pytorch image models. https : / / github . com / rwightman / pytorch - image - models, 2019. 5

    2019

  50. [58]

    Con- vnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133– 161...

    2023

  51. [59]

    Noise or signal: The role of image backgrounds in object recognition

    Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994, 2020. 1, 2, 3

    2006 arXiv

  52. [60]

    Adversarial domain adaptation with domain mixup

    Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI conference on artificial intelligence, pages 6502–6509,

  53. [61]

    Significant feature suppression and cross-feature fu- sion networks for fine-grained visual classification.Scientific Reports, 14(1):24051, 2024

    Shengying Yang, Xinqi Yang, Jianfeng Wu, and Boyang Feng. Significant feature suppression and cross-feature fu- sion networks for fine-grained visual classification.Scientific Reports, 14(1):24051, 2024. 1, 3

    2024

  54. [62]

    Improving out-of-distribution robustness via selective augmentation

    Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In International Con- ference on Machine Learning, pages 25407–25437. PMLR,

  55. [63]

    Fast segment any- thing

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment any- thing. arXiv preprint arXiv:2306.12156, 2023. 2, 3

    2023 arXiv

  56. [64]

    Ob- ject recognition with and without objects

    Zhuotun Zhu, Lingxi Xie, and Alan L Yuille. Ob- ject recognition with and without objects. arXiv preprint arXiv:1611.06596, 2016. 1, 3

    2016 arXiv

  57. [65]

    volleyball on snow

    C Lawrence Zitnick and Piotr Doll ´ar. Edge boxes: Lo- cating object proposals from edges. In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part V 13 , pages 391–405. Springer, 2014. 1, 3 12 A. Datasets ImageNet...

    2014

  58. [66]

    FULL images - the standard approach

  59. [67]

    FGC: the image is cropped according to the bounding box and padded to a square to preserve the aspect ratio

  60. [68]

    FGM: the BG is fully masked before cropping a square bounding box

  61. [69]

    BGS: BG images with shape (the FG are masked, but their shapes remain)

  62. [70]

    BGB: BG w/o shape (a minimal segmentation bounding box masks the FG) A visualization is presented in Figure 9. B.3. Combined models Here we present the fusion models in detail, including the temperature-scaled variants. We consider two fixed trained models:Φ1 and Φ2, which out...

  63. [71]

    the one with confidence ˆpi = max(ˆp1, ˆp2)

    (Max confidence) Between ˆy1 and ˆy2 choose the most confident prediction ˆyi, i.e. the one with confidence ˆpi = max(ˆp1, ˆp2)

  64. [72]

    choose the one with ˜pi = max(˜p1, ˜p2)

    (Max scaled confidence) Again we choose the more con- fident prediction ˆyi, but now the confidences are cal- ibrated using temperature scaling from Equation (2), originating from z1/T1, z2/T2, i.e. choose the one with ˜pi = max(˜p1, ˜p2)

  65. [73]

    Here t >0 is a parameter maximizing the new prediction ac- curacy on the validation set

    (Threshold prediction) We choose ˆy1 if ˆp1 > t, oth- erwise choose the higher confidence prediction. Here t >0 is a parameter maximizing the new prediction ac- curacy on the validation set

  66. [74]

    The prediction is given by argmax as usual

    (Temperature-scaled average) Let z1/T1, z2/T2 be the scaled logits vectors from Equation (2) from the two models and take the average 1 2 (σ(z1/T1) + σ(z2/T2)). The prediction is given by argmax as usual

  67. [75]

    The pose of flying birds is very different from those on the ground

    (Temperature-scaled weighted average) As before, but take a weighted average ασ(z1/T1) + (1− α)σ(z2/T2), 14 Animals like polar foxes change appearance between winter (‘snow’) and summer (‘grass’). The pose of flying birds is very different from those on the ground. Green iguan...

  68. [76]

    , z(C) 1 , z(1) 2 ,

    (Concatenate + FC layers) To model the interaction be- tween outputs of Φ1 and Φ2, we create new (train, val- idation and test) datasets by concatenating the logits for each sample x: Ψ(x) = (Φ1(x), Φ2(x)) = (z1, z2) = (z(1) 1 , . . . , z(C) 1 , z(1) 2 , . . . , z(C) 2 ) ∈ R2C...

  69. [77]

    background

    (Weighted logits combination) Generalizes the averages from confidence fusion by allowing the weights to be class-dependent vectors w1, w2 ∈ RC, representing combined logits as w1z1 + w2z2 = (w(1) 1 z(1) 1 + w(1) 2 z(1) 2 , . . . , w(C) 1 z(C) 1 + w(C) 2 z(C) 2 ). We optimize ...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.