{"id":"7c1c1931-9d62-47c0-84e3-cb50c3b92947","arxiv_id":"2412.06286","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"NADA uses a vision-language class proposer plus Stable Diffusion cross-attention maps to detect objects in paintings with no bounding box supervision, reaching 45.8 AP50 weakly supervised on ArtDL 2.0 and 21.8 AP50 zero shot.","lead":"This paper introduces NADA, a pipeline that detects objects in paintings without bounding box annotations by asking a vision-language model which classes are present and then locating those classes in Stable Diffusion attention maps. It reports the first zero-shot object detection results on two artwork datasets and beats prior weakly supervised methods on ArtDL 2.0.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Class-selectivity of Stable Diffusion attention maps is never directly tested; the detector may localize salient people rather than the queried class, so the zero-shot detection claim needs a wrong-class control.","rationale":"The strongest claim in the paper is that NADA detects specific classes in paintings using Stable Diffusion cross-attention maps, without bounding-box supervision. For that claim to hold, Eq. 2 must be class-selective: the attention map for 'Saint Peter' should mark Saint Peter, not merely the most salient person. This is the same grounding assumption the reader identified in Sec. 3.2, and it is the most load-bearing part of the pipeline because both the weakly-supervised and zero-shot variants feed the same detector; if the detector cannot distinguish classes, no class proposer can repair it. The paper's own evidence actually raises the concern. First, the qualitative example in Sec. 4.2.1 says that after misclassifying Paul as Jerome, NADA 'is still able to localize the subject,' which suggests the attention map responds to a prominent human figure regardless of which class token was used. Second, the Oracle ablation shows very low AP50 on abstract classes (angel 0.3, nudity 5.4 on IconArt, Tab. 11) while person-like classes are much higher, a pattern consistent with attention focusing on generic people rather than class-specific attributes. Third, the method averages attention over all layers and time steps (Eq. 1), which may wash out the token-specific signal that DAAM-style interpretation relies on in natural images; the paper does not re-validate this aggregation on art. None of this proves the method is wrong, but it means the central localization mechanism has not been isolated from the confounding effect of 'detect the most salient person.' The proposed control is cheap and decisive: run the same detector with wrong and generic class tokens on the same images, and measure both AP50 and mask overlap. If masks are nearly label-invariant, the headline 'first zero-shot object detection in art' is not supported by the reported experiments. Because this is an untested but testable core assumption, the conditional verdict is appropriate; the revision should add this control before the zero-shot claim can be accepted as stated.","tokens_in":14993,"tokens_out":6307,"duration_ms":68312,"concrete_test":"On a subset of ArtDL 2.0 and IconArt detection images that contain exactly one known person class, run the Oracle detector of Sec. 4.3 with three label conditions: (i) the correct class, (ii) a different person class that is absent, and (iii) the generic token 'person.' Compare AP50 of the predicted boxes and compute pairwise IoU between the attention maps Al from Eq. 2 across conditions on the same image. If the wrong-label or generic-token conditions produce boxes with AP50 comparable to the correct label, or if attention maps are highly correlated across labels, then Eq. 2 does not encode class identity and the zero-shot detection claim must be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"NADA's central claim rests on the assumption that the cross-attention map Al in Eq. 2 localizes the class token l in a painting. The paper never performs a negative control: it never runs the detector with a wrong class label on the same image and checks whether the resulting attention map still produces a confident box. The qualitative evidence in Sec. 4.2.1 points toward a failure mode: when the class proposer 'misclassifies Paul as Jerome, however it is still able to localize the subject,' the attention map tracks a salient person even though the requested class is wrong. The Oracle ablation (Sec. 4.3, Tab. 11) shows the detector is far stronger on person-like classes (Saint Sebastian 21.0, Mary 20.3) than on abstract/symbolic classes (angel 0.3, nudity 5.4), consistent with the detector latching onto generic human figures rather than class-specific iconographic attributes. If attention is not class-selective, then the AP50 numbers partly measure 'detecting a person' rather than 'detecting the queried class,' and the zero-shot detection claim overstates what is established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NADA, a no-annotation pipeline for object detection in art paintings. Given an image and a list of candidate classes, a class proposer (weakly supervised MLP on CLIP features, or zero-shot VLM prompting) selects classes, and a class-conditioned detector built from Stable Diffusion localizes them. The detector performs null-text inversion, reconstructs the image from a prompt containing the class label, averages cross-attention maps over layers and time steps (Eqs. 1-2), and applies Otsu thresholding plus watershed segmentation to obtain bounding boxes. The method is evaluated on ArtDL 2.0 and IconArt in weakly supervised and zero-shot settings, reporting AP50 of 45.8 on ArtDL 2.0 and 15.1 on IconArt for the zero-shot variant, with an Oracle upper-bound experiment isolating the detector.","tokens_in":15238,"tokens_out":3143,"duration_ms":33047,"significance":"If the central mechanism is sound, the paper would provide a training-free way to localize art-specific classes that are absent from natural-image detection datasets, which is valuable for digital humanities applications. The paper has concrete strengths: the Oracle experiment (Sec. 4.3, Tab. 11) properly isolates the detector from the class proposer; the qualitative attention maps show that the diffusion signal can align with depicted subjects; code is released; and the weakly supervised variant beats prior work on ArtDL 2.0. At the same time, the zero-shot claims rest on an unverified class-selectivity assumption, the headline numbers come from test-selected prompt variants, and the zero-shot baselines are not model-size matched. These issues are fixable with additional experiments and re-analysis, so the contribution remains promising but needs revision.","major_comments":[{"comment":"The detector's class-selectivity is not established. The paper itself notes that when the class proposer misclassifies Paul as Jerome, NADA 'is still able to localize the subject' (Sec. 4.2.1), which suggests the attention map tracks a salient person rather than the queried class. This is corroborated by the per-class Oracle results in Tab. 11: person-like classes are much higher (Saint Sebastian 21.0, Mary 20.3) than abstract/symbolic classes (angel 0.3, nudity 5.4). The authors should run a negative control in which the detector is given a wrong class label for the same image and the confidence/box output is measured; if high-confidence boxes persist under wrong-class prompts, the reported AP50 partly measures 'detecting a salient person' rather than 'detecting the queried class', and the zero-shot claim must be substantially qualified.","section":"Sec. 4.2.1 and Tab. 11"},{"comment":"Headline detection numbers are selected as the best prompt construction method per dataset (Tab. 2) and the best combination of VLM prompt and prompt construction per dataset (Tab. 4). Because this selection is done on the test split, the reported AP50 is an upper envelope over variants and is not a prediction of a single fixed pipeline. The authors should either report both variants for every setting or choose the variant on the validation split and then report the corresponding test number, with the selection rule stated before evaluation.","section":"Tabs. 2 and 4, Sec. 4.1-4.2"},{"comment":"The zero-shot comparison is confounded by model capacity: NADA's ZSCP uses LLaVA-NeXT-34B, while the InstructBLIP baseline uses Vicuna-7B, and the CLIP baseline uses ViT-B/32. Even though the same Stable Diffusion detector is used downstream, the class-proposal quality comparison and the resulting detection AP differences cannot be attributed to the prompting strategy. The authors should add a same-scale baseline (e.g., LLaVA-NeXT-7B or InstructBLIP with a comparable 34B backend) or explicitly state and justify the asymmetry as a design choice rather than an algorithmic advantage.","section":"Sec. 4.2, Tabs. 3-4"},{"comment":"The label remapping for IconArt changes the semantic content of the detection task: 'Saint Sebastien' is detected as 'person', 'child Jesus' as 'baby', and 'nudity' as 'naked person'. Consequently the per-class AP values in Tab. 11 for these classes do not measure detection of the iconographic class as defined by the dataset; they measure detection of a generic superclass. The paper should report unremapped results or explicitly reframe these classes as superclass detection and discuss how this affects the claim of detecting Christian iconography.","section":"Sec. 4, Implementation details; Tab. 11"},{"comment":"The abstract states that NADA 'outperforms prior work in weakly-supervised detection', but Tab. 2 shows that on IconArt NADA (AP50 13.8) is below Milani (16.6) and MI-Max-HL (14.5). The claim should be restricted to ArtDL 2.0 or otherwise qualified to match the reported results.","section":"Abstract and Tab. 2"}],"minor_comments":[{"comment":"There is a typo in 'no annotations for d etection in a rt' in the abstract; the spacing inside 'detection' and 'art' should be fixed.","section":"Abstract"},{"comment":"In the 'Stable Diffusion reconstruction' paragraph, the text says 'we use Stable Diffusion to generate i', but the method actually reconstructs the input image from the inverted noise; 'reconstruct' would be more precise than 'generate'.","section":"Sec. 3.2"},{"comment":"The ArtDL 2.0 'multiple' column reports AP50 of 0.03 and 0.02, which is essentially zero and strikingly low given the overall AP50 of 21.8; the authors should explain why NADA completely fails on multi-class ArtDL images, since this seems incompatible with the qualitative example of detecting Mary among five people in Fig. 4.","section":"Tab. 6"},{"comment":"There is a typo 'Crucufixion' in the text preceding Tab. 11; it should be 'Crucifixion'.","section":"Appendix D"},{"comment":"The CLIP baseline uses a cosine similarity threshold of 0.28 taken from LAION-5B filtering, but no justification is given for applying this threshold to art images; the authors should report sensitivity to this choice or justify it empirically.","section":"Sec. 4.2, baselines"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a promising direction and includes a clean Oracle isolation, but the load-bearing zero-shot claim needs a wrong-class control, the baselines need to be model-size matched, and the reported numbers need to avoid test-set selection of prompt variants. I do not see evidence of deliberate overclaiming, but the current presentation makes the zero-shot result stronger than the evidence supports. The weaknesses are addressable within the manuscript's scope, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"NADA is a genuine first: no one has reported zero-shot object detection on ArtDL 2.0 or IconArt before, and the pipeline is refreshingly simple—a VLM or trained MLP proposes classes, null-text inversion recovers Stable Diffusion cross-attention maps, and watershed segmentation turns those into boxes. The oracle ablation (Sec. 4.3) is the right kind of honesty: with perfect class labels AP50 jumps to 61.3 on ArtDL and 18.7 on IconArt, showing the detector is doing real localization and the bottleneck is class proposal. Code is promised.\n\nThe soft spots are real, and the stress-test concern lands. In Sec. 4.2.1 the authors write that when the proposer misclassifies Paul as Jerome, the detector 'is still able to localize the subject'—that is the symptom of a detector latching onto a salient person regardless of the token. The per-class oracle results fit: Saint Sebastian 21.0, Crucifixion 45.8, but angel 0.3, nudity 5.4. A wrong-class control—run the detector with a label absent from the image and measure how often it still emits a confident box—would settle this, and it is not there. Without it, the claim that Stable Diffusion attention maps are class-selective for iconographic attributes is not established.\n\nThe label remappings also shift the task. Converting 'Saint Sebastien' to 'person', 'child Jesus' to 'baby', and 'nudity' to 'naked person' turns the detection problem into finding concrete everyday nouns, which is exactly what Stable Diffusion already does well. The paper acknowledges this, but it undercuts the 'knowledge of art' narrative. The zero-shot comparison is not apples-to-apples either: NADA uses LLaVA-34B while the baselines use lighter CLIP and InstructBLIP-7B, which inflates the margin. And the abstract's 'outperforming prior work in weakly-supervised detection' is too strong—on IconArt NADA sits 2.8 AP50 below Milani, even if the body text is more careful.\n\nNone of this is fatal. The pipeline is practical for concrete classes, the oracle is a good diagnostic, and the zero-shot numbers are a first benchmark. But the paper needs a wrong-class control and fairer baselines before the central selectivity claim can be trusted.\n\nWho it is for: digital humanities researchers and anyone doing weakly supervised detection in art. It deserves a serious referee—the first zero-shot results and the code make it worth a revision cycle. I would not cite it for iconographic attribute detection until the selectivity issue is addressed, but it is a solid applied contribution.","headline":"Useful first zero-shot results and an honest oracle, but the class-selectivity of the diffusion detector is unproven; needs a wrong-class control.","tokens_in":799,"tokens_out":953,"would_cite":false,"duration_ms":42919,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stable Diffusion finds objects in paintings with zero annotations, using cross-attention maps as a detector.","keywords":["object detection in art","zero-shot detection","weakly-supervised detection","diffusion models","cross-attention maps","Stable Diffusion","null-text inversion","class proposer"],"falsifier":"A decisive test would be to run NADA's Oracle detector (perfect class labels) on a set of paintings where each class name is paired with a distractor class of the same visual category—for instance, 'Child Jesus' versus 'Saint John the Baptist as a child' or 'Saint Francis' versus 'Saint Dominic' in the same pose. If the attention maps for the correct class do not concentrate on the intended figure while excluding the distractor, or if the Oracle AP50 with correct labels is not substantially above a random-box baseline, the claim that diffusion attention localizes semantic classes in art would be refuted.","tokens_in":14850,"feed_emoji":"🎨","tokens_out":5015,"duration_ms":44070,"temperature":0.7,"pith_summary":"The paper introduces NADA, a pipeline for object detection in paintings that needs no bounding-box annotations and, in its zero-shot variant, no labels at all. It claims that the cross-attention maps of Stable Diffusion, obtained by inverting a painting and reconstructing it with a prompt containing a class name, highlight where that class appears, and that thresholding and segmenting those maps yields usable bounding boxes. Combined with a class proposer—a CLIP-based MLP in the weakly-supervised setting or a vision-language model in the zero-shot setting—NADA reports top weakly-supervised detection on ArtDL 2.0 (45.8 AP50) and the first zero-shot art-detection results, beating CLIP- and InstructBLIP-based class proposals with the same detector. The importance is that most art and historical image collections lack bounding-box annotations, so a pipeline that skips them could let digital humanities researchers localize iconographic objects at scale.","feed_headline":"Diffusion maps spot objects in art with zero labels","feed_subtitle":"NADA's cross-attention boxes beat trained detectors on ArtDL 2.0 and deliver the first zero-shot art detection.","key_machinery":"The load-bearing object is the class-conditioned cross-attention map of Stable Diffusion, extracted through null-text inversion and reconstruction. Specifically, the painting is inverted to noise under a prompt $p$ containing the class label $l$, then reconstructed; the cross-attention maps $\\{A'_{jk}\\}_{jk}$ between each token and image patch are averaged over all $J$ time steps and $K$ attention blocks (Eq. 1), and the maps for the label's tokens are averaged again (Eq. 2) to give $A_l$. A clamp to $[0,1]$, Otsu binarization, and watershed segmentation convert $A_l$ into masks whose bounding boxes are the detections. This machinery transfers the grounding property of diffusion attention from natural images to paintings without fine-tuning any component.","core_discovery":"On the paper's own terms, the central claim is that a frozen Stable Diffusion model can act as a class-conditioned detector for paintings: given a class label, null-text inversion produces noise that, when denoised with a prompt containing that label, yields cross-attention maps in which the label's token attends precisely to the regions of the painting depicting that class. Averaging these maps across layers and time steps (Eq. 1) and across the label's tokens (Eq. 2), then applying Otsu thresholding and watershed segmentation, produces bounding boxes without any training on the target data. The paper reports that this detector, fed by a weakly-supervised class proposer, reaches 45.8 AP50 on ArtDL 2.0, surpassing prior weakly-supervised methods, and that with a zero-shot proposer it reaches 21.8 AP50 on ArtDL 2.0 and 15.1 on IconArt, which the authors state are the first zero-shot object detection results in art. An oracle experiment with perfect class proposals lifts ArtDL 2.0 to 61.3 AP50, showing the detector's localization is sound while the class proposer is the main bottleneck.","pith_inferences":["If the grounding assumption holds across art styles, diffusion attention maps could serve as a free localization signal for weakly-supervised semantic segmentation in art, not just detection.","The class-proposer bottleneck suggests that pairing NADA's detector with stronger zero-shot VLMs or with iconography-aware prompts could push zero-shot AP substantially closer to the 61.3 oracle ceiling.","The failure patterns (misidentifying Paul as Jerome, localizing the wrong person as an angel) indicate that attention maps reliably find a person-like figure even when the identity is wrong; a crop-level verification step comparing prompt-to-image similarity could filter such false positives.","The same inversion-attention recipe may transfer to other label-scarce domains, such as medical or satellite imagery, provided the diffusion model has seen those image types."],"forward_implications":["NADA reaches 45.8 AP50 on ArtDL 2.0 in the weakly-supervised setting, outperforming prior trained detectors without training the detector itself, and stays within 2.8 AP50 of the strongest baseline on IconArt.","Zero-shot object detection in art is feasible: with a VLM class proposer, NADA achieves the first zero-shot results on both benchmarks, even outperforming some weakly-supervised methods.","With perfect class knowledge (Oracle), NADA's detection AP50 rises to 61.3 on ArtDL 2.0 and 18.7 on IconArt, indicating that the diffusion localizer is accurate and that better class proposers would directly improve detection.","The same detector, without retraining, localizes uncommon art-specific objects such as dragons, unicorns, banners, and swords across styles from Renaissance to ukiyo-e and surrealism.","Prompt construction materially affects localization, with template prompts working better on ArtDL 2.0 and caption prompts working better on IconArt, so the prompt must be chosen per dataset rather than universally."],"supporting_citations":[{"why":"Supplies the null-text inversion step that lets Stable Diffusion reproduce an existing painting from noise.","marker":"[38]"},{"why":"Establishes that cross-attention maps from Stable Diffusion localize words in natural images, the grounding assumption NADA transfers to art.","marker":"[56]"},{"why":"Stable Diffusion is the pretrained generation model whose cross-attention maps carry art-related knowledge.","marker":"[48]"},{"why":"The closest prior pipeline (attention maps from real images via inversion) that NADA extends to object detection in paintings.","marker":"[34]"},{"why":"Supplies the ArtDL 2.0 benchmark and the weakly-supervised baseline (Milani) that NADA outperforms.","marker":"[37]"},{"why":"Supplies the IconArt benchmark and the weakly-supervised MIL baseline for comparison.","marker":"[15]"},{"why":"CLIP provides the frozen image encoder for the weakly-supervised class proposer and serves as a zero-shot baseline.","marker":"[42]"},{"why":"LLaVA is the vision-language model used for zero-shot class proposal and caption-based prompt construction.","marker":"[30]"}],"fun_headline_variants":["Zero-shot object detection in art via Stable Diffusion","Stable Diffusion finds objects in paintings with no labels","First zero-shot art object detection, powered by Stable Diffusion","No annotations needed: Stable Diffusion localizes objects in art","Diffusion model detects objects in art without any training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole detection pipeline depends on the assumption that Stable Diffusion's cross-attention maps highlight the spatial location of the queried class in a painting; this grounding property is inherited from natural images and is not independently verified for art styles or abstract classes.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot object detection in art via Stable Diffusion","Stable Diffusion finds objects in paintings with no labels","First zero-shot art object detection, powered by Stable Diffusion","No annotations needed: Stable Diffusion localizes objects in art","Diffusion model detects objects in art without any training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2642,"prompt_tokens":951,"completion_tokens":1691,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":1614}},"tokens_in":567,"tokens_out":1691,"duration_ms":10599,"temperature":1.0,"reasoning_tokens":1614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:48:44.332631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would be to run NADA's Oracle detector (perfect class labels) on a set of paintings where each class name is paired with a distractor class of the same visual category—for instance, 'Child Jesus' versus 'Saint John the Baptist as a child' or 'Saint Francis' versus 'Saint Dominic' in the same pose. If the attention maps for the correct class do not concentrate on the intended figure while excluding the distractor, or if the Oracle AP50 with correct labels is not substantially above a random-box baseline, the claim that diffusion attention localizes semantic classes in art would be refuted.","supporting_citations":[{"cited_title":"Null-text inversion for editing real images using guided diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the null-text inversion step that lets Stable Diffusion reproduce an existing painting from noise."},{"cited_title":"What the daam: Interpreting stable dif- fusion using cross attention","cited_arxiv_id":null,"evidence_quote":"Establishes that cross-attention maps from Stable Diffusion localize words in natural images, the grounding assumption NADA transfers to art."},{"cited_title":"Proposals generation for weakly supervised ob- ject detection in artwork images","cited_arxiv_id":null,"evidence_quote":"Supplies the ArtDL 2.0 benchmark and the weakly-supervised baseline (Milani) that NADA outperforms."},{"cited_title":"Weakly supervised object detection in artworks","cited_arxiv_id":null,"evidence_quote":"Supplies the IconArt benchmark and the weakly-supervised MIL baseline for comparison."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"CLIP provides the frozen image encoder for the weakly-supervised class proposer and serves as a zero-shot baseline."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"LLaVA is the vision-language model used for zero-shot class proposal and caption-based prompt construction."}],"review_version":1}