{"id":"29715ac7-b341-4391-a048-0ab18badfdc8","arxiv_id":"2412.00095","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An image captioning model that injects detector-supplied object labels and predicted attributes reduces object hallucination, but its quality gains are inconsistent and become negative on strong CLIP encoders.","lead":"OPCap is a captioning method that feeds detected object labels and predicted attributes into the decoder to reduce object hallucinations. Tests on COCO and nocaps show fewer hallucinated objects, but standard caption quality metrics improve only for weak image encoders and fall for the strongest CLIP backbone.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CHAIR gains are partly by construction because YOLOS-tiny uses the same COCO object vocabulary that CHAIR scores against; the central hallucination claim is not independently established.","rationale":"The reader's weakest assumption was detector accuracy/completeness, which is a real limitation but not the most load-bearing one: even a perfect detector would still produce CHAIR gains inflated by the vocabulary overlap between the detector and the metric. The reader did note in the rationale that the CHAIR gain is partly by construction, but did not elevate it to the primary concern. The method's strongest and most consistent evidence is the CHAIR improvement, yet that evidence is confounded. Meanwhile, the quality claim is directly contradicted on COCO for CLIP-large (CIDEr and SPICE drop), and the self-defined CLIP Vote metric is unvalidated and does not even favor +OP for CLIP-large. Conditional acceptance is still appropriate because the core idea is testable and the proposed checks could substantiate it, but the reported hallucination reduction should not be accepted as established until a vocabulary-disjoint evaluation is performed. I found no indication of intentional misrepresentation; the issue is an unexamined methodological confound and overclaimed conclusions.","tokens_in":1088,"tokens_out":713,"duration_ms":82240,"concrete_test":"Re-evaluate CLIP-large and CLIP-large+OP on COCO using a hallucination metric that does not use the COCO 80-class vocabulary as its object checklist, e.g., CHAIR-style scoring with ground-truth object labels from Visual Genome, OpenImages, or human presence judgments on a random sample of 200 captions. If the +OP CHAIR advantage disappears or reverses, the reported hallucination reduction is a vocabulary-overlap artifact rather than genuine grounding. A cheap analytic check is to compute the overlap between YOLOS-tiny's 80 classes and CHAIR's object-word list; near-total overlap confirms the confound.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that OPCap \"effectively mitigates hallucination,\" supported mainly by CHAIRs/CHAIRi improvements in Table 2b. But CHAIR evaluates hallucination by checking whether generated object words appear in the image's COCO ground-truth objects (Ref [13]), while YOLOS-tiny is trained on COCO's 80 classes (Section 3.1, Ref [17]). Thus the detector's output vocabulary is almost identical to the vocabulary CHAIR uses to decide what counts as a hallucinated object. Injecting COCO-class object labels into the decoder therefore makes it easier to avoid CHAIR-detected hallucinations, even if the injected labels are wrong for the image. This is consistent with Table 2b: CLIP-large+OP improves CHAIR but degrades CIDEr (117.39 to 115.59) and SPICE (21.12 to 20.8). Qualitative examples also show artifacts like \"a statue of a statue\" and \"a plane flying over an airplane.\" Because the hallucination metric shares the detector's vocabulary, the reported CHAIR improvements are not independent evidence that the captions are more factually grounded.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"OPCap is a prompting strategy for image captioning that augments a frozen image encoder and a Transformer decoder with object-level information: a YOLOS-tiny detector supplies object labels and boxes, a CLIP-based attribute predictor supplies per-object attributes from cropped regions, and the resulting [OBJ]/[ATTR] token sequence is concatenated with image features as the decoder's key/value input. The authors claim that this reduces object hallucination and significantly improves caption quality on COCO and nocaps, that the method applies to any image captioning model, and that it requires no additional pretrained models or large-scale datasets. The experiments compare EffNet-, CLIP-base-, and CLIP-large-based captioners with and without the 'OP' intervention on CIDEr, BLEU-4, METEOR, ROUGE-L, SPICE, CHAIRs, CHAIRi, and a newly proposed CLIP Vote protocol. The clearest positive result is for the weak EffNet encoder, where all quality and hallucination metrics improve; the results for the CLIP-based models are mixed or negative.","tokens_in":8721,"tokens_out":13988,"duration_ms":111205,"significance":"If the claims survive revision, the contribution is useful: a lightweight, interpretable intervention (detected objects plus attributes as decoder prompts) that produces large gains for a weak encoder (EffNet: CIDEr 55.95 to 68.09, CHAIRs 40.84 to 32.78) and consistent CHAIR improvements across all tested models. The method is deliberately simple, needs no language model, and the failure mode on CLIP encoders is acknowledged in Section 4.3, which is honest. The paper is not publishable in its present form, however: the abstract's quality claim is contradicted by Table 2b, the claimed ablations are absent, the hallucination gains are confounded by vocabulary sharing between YOLOS-tiny and the CHAIR metric, and the implementation details are insufficient for reproducibility. These problems are fixable in revision, so a major revision is the appropriate outcome.","major_comments":[{"comment":"The abstract's claim that OPCap 'significantly improves the quality of generated captions' is contradicted by the paper's own Table 2b: for CLIP-large+OP, BLEU-4 (36.64 to 35.96), METEOR (28.21 to 27.47), ROUGE-L (56.93 to 56.52), CIDEr (117.39 to 115.59), and SPICE (21.12 to 20.80) all decrease, and Table 2a shows large out-of-domain drops on nocaps (CIDEr 57.14 to 47.02). Section 4.3 itself states that 'models based on CLIP show the opposite trend,' so the unqualified statements in the abstract and conclusion should be replaced by a model-regime-specific claim, with the strong-encoder degradation discussed explicitly as a limitation.","section":"Abstract / Section 4.3 / Table 2b"},{"comment":"The hallucination-mitigation evidence is confounded and partially contradicted. CHAIR (Ref [13]) penalizes object words that are absent from the COCO ground-truth object list, and the detector YOLOS-tiny is trained on the same COCO 80-class vocabulary; injecting detector labels into the decoder therefore moves CHAIR in the expected direction partly by construction, even when the injected labels are wrong for the image. Table 1's CLIP-large+OP captions contain new artifacts ('a statue of a statue on a counter', 'a plane flying over an airplane in a desert') that are precisely the hallucination type the paper claims to reduce, and Section 4.2 itself concedes that the strategy is partially dependent on detector performance. A concrete test is needed, for example evaluating hallucination against a vocabulary disjoint from the detector's training classes, or analyzing CHAIR gains conditioned on detector correctness, to show that the gains are not an artifact of vocabulary alignment.","section":"Section 3.1 / Table 2b / Section 4.2 / Table 1"},{"comment":"The introduction states 'We conducted experiments and ablation studies,' and the conclusion attributes the gains to 'the target-aware strategy and the fusion module,' but no ablation experiment appears anywhere in Section 4. In particular, there is no experiment isolating the attribute predictor (object labels alone versus labels plus attributes) or the caption token dropout, so the paper's central design claim, that attribute prediction contributes to the reported gains, is not demonstrated.","section":"Section 1 / Section 5"},{"comment":"The method is not reproducible as specified: the top-k in Section 3.1, the sigmoid threshold for attribute selection, the caption token dropout rate, and the object detector confidence threshold are never given values; no training hyperparameters (learning rate, epochs, batch size, optimizer, decoder initialization) are reported; the COCO split description in Section 4.1 ('118,287 images... 5,000 images set aside for validation and testing') does not identify the split used for the captioning evaluation; and the nocaps protocol (validation set versus evaluation server) is unspecified. In addition, no error bars or significance tests accompany the comparisons, several of which are small, such as the CLIP-large CHAIRi change from 4.47 to 4.27.","section":"Section 3.1 / Section 4.1 / Section 4.3"},{"comment":"Table 3 and its surrounding text are internally inconsistent. The text claims that 'models incorporating the OPCap method consistently received more votes,' but CLIP-large+OP (1093) receives fewer votes than CLIP-large (1250); the sentence claiming that 'the base version of CLIP received the highest number of votes' is also contradicted by the table, in which CLIP-base+OP (1391) is highest. Moreover, since CLIP is used both as the scoring model and inside the OPCap attribute predictor, and two of the baselines are themselves CLIP-based captioners, this no-reference metric is biased toward the method's own components and does not independently corroborate the quality claim.","section":"Table 3 / Section 4.3"}],"minor_comments":[{"comment":"The heading 'Hallucnation in models' in Section 2.2 and the heading 'Evualation Metrics' in Section 4.1 contain typos.","section":"Section 2.2 / Section 4.1"},{"comment":"The rows for AoA and UpDown appear misaligned: the values 27.4 and 26.9 sit in the ROUGE-L column but are in the plausible range of METEOR (compare 27.09 for ViTGPT2 and 26.72 for CLIP-base), which would leave ROUGE-L unreported for these baselines.","section":"Table 2b"},{"comment":"The passage 'This aligns with the findings in []' contains an empty citation that should be filled or removed.","section":"Section 4.3"},{"comment":"Reference hygiene is incomplete: Refs [2] and [25], Refs [4] and [26], and Refs [19] and [27] are the same papers listed twice, and Refs [1] and [13] are also the same work with different years.","section":"References"},{"comment":"The contribution bullet in Section 1 claims the method works 'without requiring additional pre-trained models,' which is contradicted by Section 3.1's use of the pre-trained YOLOS-tiny detector and a pre-trained CLIP encoder inside the attribute predictor.","section":"Section 1 / Section 3.1"},{"comment":"Equation (4) states that the object and attribute tokens map to 2 x (o + k) feature vectors, but the Section 3.1 example with k = 2 attributes per object produces o object tokens plus 2o attribute tokens plus special tokens, which does not equal 2 x (o + k); the token accounting and the role of the [OBJ] and [ATTR] special tokens need to be clarified.","section":"Equation (4) / Section 3.1"},{"comment":"The claim that 'both the object detector and attribute extractor are trained or fine-tuned on the COCO dataset' conflicts with Section 3.1, which states that the attribute predictor is trained on the VAW dataset [18].","section":"Section 2.3"},{"comment":"The related-work discussion implies that OPCap 'can be applied to any image captioning model,' but only EffNet and CLIP-based variants are evaluated with +OP, and Section 3.2 restricts the decoder to a Transformer because 'LSTM has difficulty handling the joint features,' so the universality claim is neither tested nor supported by the design.","section":"Section 3.2 / Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a short conference paper (the related work and Table 3 cite WACV/Fusecap), and the claims in the abstract and conclusion are broader than the evidence in the tables. The contribution list ('applies to any model', 'no additional pre-trained models') is contradicted by the method section, and the novelty boundary with respect to Biten et al. [15] rests entirely on the attribute predictor, which is precisely the component that has no ablation. I would recommend that the editor require a reproducibility statement or code release, and insist on the missing ablations before the paper is reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest, conditional contribution. The genuinely new bit is combining detected object labels with predicted attributes as prompt tokens for a caption decoder; the cited work [15] only injects labels. That narrow extension is worth a look, and the method is resource-efficient and model-agnostic in spirit. The paper also gives credit where due: it openly states the detector dependency in Section 4.2 and shows a failure case where missed objects lead to omissions.\n\nThe empirical support is the soft spot. The paper claims ablations but reports none; the closest prior baseline, [15], is never compared directly. On the strongest backbone, CLIP-large on COCO, the caption quality metrics drop after adding OPCap (CIDEr 117.39 to 115.59, SPICE 21.12 to 20.8). The abstract's claim of \"significantly improves the quality\" is simply not supported for that model. The CHAIR improvements are real in the table, but the stress-test note is right: YOLOS-tiny is trained on COCO's 80 classes, and CHAIR evaluates against COCO ground-truth objects, so the detector and the hallucination metric share essentially the same vocabulary. Injecting detected labels into the decoder makes it easier to avoid CHAIR-detected hallucinations, even if the labels are imperfect. That doesn't make the method useless, but it means the central hallucination claim is not independently established. The qualitative examples also show artifacts like \"a statue of a statue\" and \"a plane flying over an airplane,\" which undercut the quality narrative.\n\nOther issues: no error bars or significance tests, vaguely described dataset split (5,000 images for both validation and testing?), and a missing citation placeholder in the discussion of hallucination metrics. The free parameters (top-k, sigmoid threshold, token dropout, detector confidence) are not ablated.\n\nWho is this for? Researchers working on lightweight hallucination fixes in captioning might find the attribute-prompting idea a useful starting point. But as it stands, the paper needs substantial revision: real ablations, a direct comparison with [15], error bars, and an explicit discussion of the CHAIR-vocabulary alignment. I'd send it to review, not desk-reject, because the core idea is testable and the direction is sensible. But it's a conditional accept at best, and the authors should be pushed hard on the metric issue and the missing comparisons.","headline":"A plausible but thinly evidenced extension of object-conditioned captioning; the CHAIR gains are partly by construction, and the quality claim fails on the strongest backbone.","tokens_in":9254,"tokens_out":1515,"would_cite":false,"duration_ms":15542,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that feeding a captioning decoder a list of detected objects and their predicted attributes reduces object hallucination in generated captions.","keywords":["image captioning","object hallucination","object detection","attribute prediction","target-aware prompting","transformer decoder","COCO","nocaps"],"falsifier":"Take a set of COCO images whose ground-truth captions mention objects that YOLOS-tiny is known to miss, generate captions with and without OPCap, and count whether the missed objects vanish from captions and whether new false objects appear. As a cleaner test, erase all detector outputs on a validation batch (feeding OPCap an empty object list): if hallucination stays at the base model's level, the detector's object list is the carrier of the effect.","tokens_in":8264,"feed_emoji":"🖼️","tokens_out":4995,"duration_ms":40212,"temperature":0.7,"pith_summary":"This paper argues that image captioning models invent objects because the decoder is never told which objects are actually present. OPCap adds a target-aware prompting stage: an object detector returns labels and bounding boxes, the boxes are cropped, and an attribute predictor labels each crop with properties such as color, shape, and state. Those labels and attributes are written as delimited token sequences and fed into a Transformer decoder together with the image features. The paper reports lower CHAIRs and CHAIRi hallucination scores when OPCap is added to EffNet, CLIP-base, and CLIP-large captioners, and higher traditional caption metrics for the weaker EffNet encoder. The broader claim is that explicit object-level grounding can reduce hallucination without extra pretrained language models or large-scale data.","feed_headline":"Object-aware prompts cut hallucinated captions","feed_subtitle":"A detector and attribute predictor tell the decoder what is genuinely in the image, lowering CHAIR scores on COCO and nocaps.","key_machinery":"The load-bearing mechanism is the object-aware prompt: a sequence of special tokens ([OBJ] label, [ATTR] attribute) produced by a YOLOS-tiny detector and a CLIP-plus-MLP attribute predictor trained on the VAW dataset. These tokens are embedded into the same d_model space as the image features and prepended to them, giving the Transformer decoder an explicit, current inventory of objects and their properties. The mechanism supplies the decoder with information the image encoder may have compressed or lost, so caption generation is conditioned on a direct statement of what is present.","core_discovery":"The central claim is that a target-aware prompting strategy, OPCap, reduces object hallucination in image captioning by injecting detected object labels and their predicted attributes into the decoding process. Given an image, a pretrained object detector returns labels and boxes; the boxes crop regions that an attribute predictor labels with properties such as color, shape, and state. These are formatted as [OBJ] ... [ATTR] ... token sequences, embedded, and concatenated with the image encoder's features to form the Transformer decoder's cross-attention memory. Training freezes the image encoder and requires only the language-model captioning loss. The paper reports lower CHAIRs and CHAIRi scores after adding OPCap across the EffNet, CLIP-base, and CLIP-large decoders, and higher CIDEr and SPICE for the weaker EffNet encoder; it interprets the overall pattern as evidence that explicit object grounding without extra language models makes captions describe what is actually in the image.","pith_inferences":["Editorial inference: the benefit concentrates on weaker image encoders, so a natural extension is to make the prompt adaptive, injecting object tokens only when the encoder's own features are uncertain rather than always.","Editorial inference: because the attribute predictor is trained on the VAW vocabulary, its label set may not cover attributes that captions actually need; comparing OPCap under different attribute vocabularies would test whether the vocabulary is the bottleneck.","Editorial inference: the discrete token prompt is a lossy bottleneck; a dense region-feature conditioning variant might keep the hallucination reduction while avoiding the traditional-metric regression seen on CLIP-large."],"forward_implications":["Any existing encoder-decoder captioning model can be retrofitted with OPCap by adding the detector and attribute predictor, freezing the encoder, and training only the decoder with the standard language-model loss.","Resource-constrained deployments can attack hallucination without loading a large language model, because the added components are lightweight (YOLOS-tiny and a small CLIP-based classifier).","The paper observes that adding OPCap to CLIP-large lowers CIDEr while improving CHAIR, so evaluations aimed at grounded captions should report hallucination metrics alongside traditional caption metrics.","On nocaps out-of-domain images, the weak EffNet encoder improves on all reported metrics with OPCap, suggesting explicit object grounding helps describe unseen categories when the base encoder is limited."],"supporting_citations":[{"why":"Supplies the object detector (YOLOS-tiny) that produces the labels and boxes forming the prompt.","marker":"[17]"},{"why":"Provides the VAW attribute vocabulary and training data for the attribute predictor.","marker":"[18]"},{"why":"CLIP serves as the image encoder for the attribute predictor and as the scoring model in the CLIP Vote evaluation.","marker":"[19]"},{"why":"Defines the CHAIR hallucination metrics used as the paper's primary evidence of reduced hallucination.","marker":"[13]"},{"why":"MSCOCO is the main training and evaluation dataset for the captioning experiments.","marker":"[20]"},{"why":"nocaps provides the out-of-domain generalization evaluation.","marker":"[28]"}],"fun_headline_variants":["OPCap grounds captions in real objects and attributes","Target-aware prompt kills object hallucination","Detector + attributes = truthful image captions","Object labels and attributes anchor caption generation","OPCap: object-aware prompting stops caption hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"If the object detector misses an object or misclassifies it, OPCap cannot correct the error: the omitted object will not appear in the caption, and a wrong label can inject an attribute from the wrong region.","fun_headline_variants_meta":{"raw":{"variants":["OPCap grounds captions in real objects and attributes","Target-aware prompt kills object hallucination","Detector + attributes = truthful image captions","Object labels and attributes anchor caption generation","OPCap: object-aware prompting stops caption hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000609,"raw_usage":{"total_tokens":2782,"prompt_tokens":838,"completion_tokens":1944,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":1876}},"tokens_in":454,"tokens_out":1944,"duration_ms":13110,"temperature":1.0,"reasoning_tokens":1876,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:11:30.611268+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of COCO images whose ground-truth captions mention objects that YOLOS-tiny is known to miss, generate captions with and without OPCap, and count whether the missed objects vanish from captions and whether new false objects appear. As a cleaner test, erase all detector outputs on a validation batch (feeding OPCap an empty object list): if hallucination stays at the base model's level, the detector's object list is the carrier of the effect.","supporting_citations":[{"cited_title":"Learning to predict visual attributes in the wild","cited_arxiv_id":null,"evidence_quote":"Provides the VAW attribute vocabulary and training data for the attribute predictor."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"CLIP serves as the image encoder for the attribute predictor and as the scoring model in the CLIP Vote evaluation."},{"cited_title":"Lawrence Zit- nick, and Piotr Doll ´ar","cited_arxiv_id":null,"evidence_quote":"MSCOCO is the main training and evaluation dataset for the captioning experiments."},{"cited_title":"no- caps: novel object captioning at scale","cited_arxiv_id":null,"evidence_quote":"nocaps provides the out-of-domain generalization evaluation."}],"review_version":1}